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Executive Summary 


Opportunity Works was a three-year effort led by Jobs for the Future to help 
opportunity youth—young people ages 16 to 24 who are not in school or meaningfully 
employed—access postsecondary and career pathways. Based on the Back on Track 
framework, seven cities across the country undertook collective impact approaches 
with diverse partners to provide supportive, enhanced preparation and 
postsecondary/career bridging for eligible young people, with a particular focus on 


young men of color. 


A quasi-experimental evaluation conducted by the Urban Institute in three 
Opportunity Works sites found large, consistent, positive effects on participants’ 
postsecondary enrollment and increased connection with either education or 
employment about one year after program entry. Specifically, Opportunity Works 
participants were twice as likely to enroll in college and 25 percent more likely to be in 
either education or employment. Postsecondary results were even greater for young 
men of color, who were nearly six times as likely to enroll in college. This report also 


includes insights and lessons from qualitative field research. 


What Is Opportunity Works? 


Between the critical ages of 16 and 24, many low-income youth risk becoming disconnected from 
school and the labor market. In 2016, 6 percent of young people ages 16 to 24 were not in school and 
did not have high school credentials; among youth from the lowest-income families, nearly 10 percent 
disconnected from school.t Males are about 45 percent more likely to drop out than females.2 About 
14 percent of high school dropouts in this age range are unemployed and about 40 percent are not in 
the labor force.? But the employment prospects for opportunity youth are encouraging if they gain the 


necessary postsecondary credentials and skills. 


The Back on Track framework fosters the growth and scale of programs aimed at improving the 


postsecondary success of opportunity youth. Back on Track is characterized by three program phases: 


= Enriched preparation: recruits high school noncompleters ages 16 to 24 and provides them 
with the curriculum, support, and coaching essential for educational success and career 


readiness, as well as support in completing a high school equivalency (HSE). 


= Postsecondary/career bridging: helps students bridge to college and/or careers. This phase 
caters to opportunity youth who already have or are very close to obtaining high school 


credentials and helps them build the skill set essential for postsecondary achievement. 


= First-year support: encourages program staff members to continue working with students 


through their first year of college or career. 


The impact evaluation of Opportunity Works covered three sites that focused on the 
postsecondary and career-bridging phase of the model: Hartford, CT; Philadelphia, PA; and South King 
County/Seattle, WA. In addition, Boston, MA; New Orleans, LA; San Francisco, CA, and Santa Clara 
County, CA, participated in Opportunity Works and were included in an in-depth implementation 


evaluation. 


Figure 1 shows the logic model for this phase, as it was specified for this initiative. Since 
Opportunity Works launched, Jobs for the Future has refined the framework, but the original 2014 


approach is the subject of this evaluation. 
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FIGURE 1 


Back on Track Postsecondary Bridging Logic Model (Original 2014 Framework) 
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Short-Term Outcomes 

Soft Skills for Success 

* Students gain postsecondary & career navigation skills 

* Students exhibit career-ready skills & behaviors 

Access College/Career Coursework 

* Students have access to postsecondary facilities 

* Students enrollin credit-bearing courses appropriate to 
reinforcing essential skills 

Subject Knowledge Gains 

* Students demonstrate proficiency in key skills & subject areas 

Support Systems 

* Students havea sufficient care network 

* Students build social networks supportive of postsecondary 
success 


Medium-Term Outcomes 
Confirmatory 
* Students apply to training programs, including college 
* Students enrollin training programs, including college 
* Students earn college credits 
* Students enroll for more than one semester 
* Student enroll for more than one year 
* Students earn college or industry-recognized credentials 
* Students complete 12 credits of college work 
Exploratory 
* Students do not experience subsequent arrests or incarceration 


Long-Term Outcomes 
Students are not disconnected from work and education 
Students become employed 
Students are employed in quality jobs 
Students’ earnings are higher than they would be otherwise 


Contextual factors: Local labor market, service infrastructure for youth, juvenile justice system, availability of adult and postsecondary education options, policies 
affecting youth (e.g., GED changes, Pell Grant availability) 


Did Opportunity Works Help Youth? 


The Urban Institute-led evaluation measured the effects on Opportunity Works participants in 


Hartford, Philadelphia, and South King County/Seattle. Participant outcomes were compared with 


those of opportunity youth in other programs not governed by the Back on Track framework. For the 


analysis, our study uses both the National Student Clearinghouse (NSC) survey and a follow-up 


Opportunity Works survey, as well as data from a baseline survey. 


Opportunity Works programs’ effects on postsecondary enrollment were strongly positive (figure 


2). When pooling the sites together, Opportunity Works participants are more likely to enroll in 


postsecondary institutions than their matched counterparts. About 53 percent of program participants 


enrolled in a postsecondary institution after the baseline survey, compared with only 26 percent in the 
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matched comparison group. In other words, participants were more than twice as likely as similar 


youth not in Back on Track programming to have enrolled in a postsecondary institution. 


Further, program participation was associated with higher postsecondary enrollment in all sites. 
Opportunity Works participants in Hartford were 25 percentage points more likely to enroll ina 
postsecondary institution than their counterparts in the matched comparison group (a 93-percent 
gain), 12 percentage points more likely to enroll in Philadelphia (a 40-percent gain), and 44 percentage 


points more likely to enroll in South King County (a 200-percent gain). 


FIGURE 2 
Impact of Opportunity Works on College Enrollment 
Share of respondents enrolled in any postsecondary institution 


S Treatment Matched comparison 


67.4%*** 


55.05" 52.0%"** 


40.2%* 


All sites Hartford Philadelphia South King County 


Sources: National Student Clearinghouse and Opportunity Works baseline surveys. 
Note: Differences in means between the treatment and control groups are statistically significant at *** p < 0.01 and ** p < 0.1. 


The program also had positive effects on several other education outcomes (figure 3). Program 
participants were about 50 percent more likely to have applied to an associate’s degree program than 
people in the matched comparison group (43 versus 29 percent). Opportunity Works participants 
were about 40 percent more likely to apply to training programs than people in the comparison group 
with similar baseline characteristics (25 versus 18 percent). Program participants were almost twice as 
likely to complete a Free Application for Federal Student Aid (FAFSA) than the matched comparison 


group (71 versus 37 percent). 
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Opportunity Works participants were also less likely to be disconnected from work and education. 
About 21 percent of Opportunity Works participants were not enrolled in school or working, 


compared with 46 percent of the comparison group (a 53 percent reduction). 


FIGURE 3 
Impact of Opportunity Works on Other Education and Labor Market Outcomes 


| Treatment Matched comparison 


70.9%*** 


43.1%*** 


24.9%*** 


21.4%*** 


Applied to an associate's Applied to a training program Completed FAFSA Not working and not 
degree program enrolled in school 


Sources: Opportunity Works baseline and follow-up surveys. 
Note: Differences in means between the treatment and control groups are statistically significant at *** p < 0.01. 


While the program had large effects on education and labor outcomes, no significant program 


effects appeared for exploratory outcomes, such as college graduation goals and the likelihood of 
being arrested (figure 4). 
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FIGURE 4 
Impact of Opportunity Works on Exploratory Outcomes 


mTreatment Matched comparison 


69.7% 


Goal to graduate from a two-year Goal to graduate from a four-year Arrested since baseline 
program program 


Sources: Opportunity Works baseline and follow-up surveys. 


What Lessons Can Be Learned from Opportunity 
Works? 


The magnitude of the impact of Opportunity Works on postsecondary enrollment is large, more than 
doubling overall enrollment. These findings are consistent across the three diverse sites in the impact 
study, which suggests that the Back on Track postsecondary bridging framework can be credited for 


the positive effect. 


The Back on Track postsecondary bridging framework also appears to have helped participants 
reduce their chance of disconnection from education and employment. However, the program did not 
affect some of the exploratory outcomes around goals for education and justice involvement. It could 
be valuable to build out elements of the model that might improve these outcomes, such as mentoring 
support, which appeared in the original framework but was not implemented in any Opportunity 
Works site (Anderson et al. 2017). 
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Introduction 


The Back on Track framework describes a multiphase intervention to help opportunity 
youth—young people ages 16 to 24 who are not in school or meaningfully employed— 
gain access and succeed in pathways to postsecondary education and training and 
careers. Opportunity youth have been a growing focus of poverty alleviation, workforce 
development, and social inclusion efforts across the country. In early 2015, Jobs for the 
Future (JFF), in collaboration with the Aspen Institute Forum for Community Solutions, 
contracted the Urban Institute to evaluate a seven-city demonstration of the signature 
Back on Track framework. The demonstration was sponsored by a grant from a Social 
Innovation Fund (SIF) and national match funders, and branded Opportunity Works. 
This report summarizes findings from the in-depth implementation study, reported in 
more detail in an earlier publication, and describes the methods and findings from the 
impact study of the Opportunity Works intervention at the program sites. It is the final 


public report from the evaluation effort. 


Opportunity Youth 


Between the critical ages of 16 and 24, many low-income youth risk becoming disconnected from 
school and the labor market. In 2016, 6 percent of young people ages 16 to 24 were not in school and 
did not have high school credentials; among youth from the lowest-income families, nearly 10 percent 
were disconnected from school.* Males are about 45 percent more likely to drop out than females. * 
About 14 percent of high school dropouts in this age range are unemployed and about 40 percent are 
not in the labor force.® A major contributor to lower labor force participation is a lack of high school or 
postsecondary credentials, labor market experience, and other forms of human capital. Consequently, 


social interventions can have an important effect on the outcomes of these opportunity youth. 


The national and global economies have high demand for an educated labor force. Within the US 
labor market, jobs that require at least some postsecondary education are projected to increase 
substantially within the next 10 years.’ In this sense, the employment prospects for opportunity youth 


are encouraging if they gain the necessary postsecondary credentials and skills. 


The Back on Track Framework 


The Back on Track framework fosters the growth and scale of programs aimed at improving the 
postsecondary success of opportunity youth. Back on Track is characterized by three program phases, 
each with its own set of features. These are designed to help opportunity youth move toward 


postsecondary and career success. 


= The first phase, enriched preparation, recruits high school noncompleters ages 16 to 24 and 
provides them with the curriculum, support, and coaching essential for educational success 
and career readiness. This phase helps students complete their high school equivalency (HSE) 
or diploma and focuses on creating a college culture through reinforcing interactions and 
procollege physical spaces, offering college- and career-ready curricula and instruction and 


customized and accelerated instruction, and providing personalized guidance and support. 


= The second phase, postsecondary/career bridging,® helps students bridge to college and/or 
careers through supported dual enrollment, sharing of college knowledge and success 
strategies, personalized guidance with connections to “best bets,” mentorship from program 
graduates, and supported transition to college. This phase caters to opportunity youth who 
already have or are very close to obtaining high school credentials and helps them build the 
skill set essential for postsecondary achievement. In this context, “postsecondary” refers to 
both academic-track college courses and career-oriented professional training courses offered 


at the postsecondary level by higher education institutions or other training providers. 


= The final phase, first-year support, encourages staff members to continue working with 
students through their first year of college or career, particularly focusing on developing an 


attachment to postsecondary education. 


JFF required each subgrantee site to choose one of the first two phases (enriched preparation or 


postsecondary bridging) as a primary focus of their Opportunity Works intervention for the SIF grant. 
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FIGURE 5 
Back on Track Enriched Preparation Logic Model (Original 2014 Framework) 


Contextual factors: Local labor market, service infrastructure for youth, juvenile justice system, availability of adult and postsecondary education and training 
options, policies affecting youth (e.g., GED changes, Pell Grant availability) 
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FIGURE 6 
Back on Track Postsecondary Bridging Logic Model (Original 2014 Framework) 


Contextual factors: Local labor market, service infrastructure for youth, juvenile justice system, availability of adult and postsecondary education options, policies 
affecting youth (e.g., GED changes, Pell Grant availability) 
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Both the enriched preparation and postsecondary bridging phases of Back on Track help at-risk 
youth achieve postsecondary success while simultaneously fulfilling the labor market needs of the 
national and global economy, but they are designed to serve youth at different stages of school 
progress and career readiness. Thus, the overarching logic models for the two stages of Back on Track 
have similar features, as demonstrated in figure 5 and figure 6. In particular, the long-term outcomes 
of both Back on Track stages are the same, but the enriched preparation participants would need a 
longer time horizon and different intervening service components to meet them. In addition, in the 
ideal structure of the enriched preparation model for Back on Track, participating youth will go on to 
participate in the postsecondary bridging model and receive first-year support to continue their 
academic and career journey. However, JFF designed the features so they could stand in isolation if 


necessary. 


Back on Track served as a framework that programs could use to structure their service delivery 
to help young people obtain secondary, postsecondary, and employment success. JFF purposefully 
conveyed Back on Track as a framework rather than a rigid model, and individual programs could 
adapt Back on Track to their local context. Though JFF introduced further refinements to the Back on 
Track framework in late 2016, this report considered the original framework based on JFF (2014)—the 


model the sites used at the time of data collection. 


The Evaluation 


JFF received a grant from the Corporation for National and Community Service's SIF program to 
implement Back on Track in Boston, MA; Hartford, CT; Philadelphia, PA; New Orleans, LA; San 
Francisco, CA; Santa Clara County, CA; and South King County/Seattle, WA. This effort is known as 
“Opportunity Works.” Since the Corporation for National and Community Service requires rigorous, 
independent evaluation of funded activities, JFF hired the Urban Institute as a third-party evaluator to 


document implementation and assess the effects of Back on Track programs. ? 


The evaluation takes a mixed-methods research approach across the seven sites. !° This report 
summarizes key findings from the implementation study, which is documented in detail in Anderson et 
al. (2017); presents new findings from the impact study; and provides reflections on the overall 
effectiveness of the Back on Track model to help opportunity youth achieve education and other 
quality-of-life outcomes. The program impact results come from a quasi-experimental impact study 
reviewed and approved by the Corporation for National and Community Service. Further details about 


the study design appear in the “Impact Study Methods” section. 
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Program Operations 


Site Profiles 


The following summaries provide high-level details of each site’s program operations to contextualize 
this report’s findings. More detailed descriptions of the site’s program efforts can be found in the Site 


Summary Appendix from the interim project implementation report. 


Boston, Massachusetts 
Back on Track approach: Postsecondary/career bridging 


Target population: Young people ages 20 to 24 with high school credentials who are not yet 
connected to postsecondary educational opportunities or employment 


Number of young people served through August 2017: 377 


Impact evaluation site: No 


PROGRAM DETAILS 

In 2013, the Boston Private Industry Council (PIC) established a Connection Center, the hub of youth 
outreach and recruitment for Opportunity Works. The Connection Center was originally located in a 
“T” train station to make it accessible to young people across the city. The Connection Center helped 
find and connect youth to postsecondary opportunities through a set of contracted partners. In 
collaboration with these partners, the Connection Center helped youth understand and explore their 
career options, provided coaches to support them, and aided them in applying for financial aid and 
various training or college programs. Additionally, the Connection Center and contracted partners 


assisted youth to overcome barriers in the process. 


OUTREACH AND RECRUITMENT METHODS 


= Posting flyers with contact information for the Connection Center 


PRIMARY REASONS FOR ATTRITION 


= Housing 


= Food insecurity 


PARTNERSHIPS 
Collective impact effort: Boston Opportunity Youth Collaborative 


Backbone organization: Boston PIC (subgrantee) and Boston Opportunity Agenda 
Primary partners: 


= X-Cel Education 

= Asian American Civic Association 
= College Bound Dorchester 

= Inquilinos Boricuas en Accién 


= Jewish Vocational Services 


Hartford, Connecticut 
Back on Track approach: Postsecondary/career bridging 


Target population: Young people ages 16 to 24 with or without high school credentials who are 
unemployed and/or not enrolled in postsecondary education, especially young men of color 


Number of young people served through August 2017: 108 


Impact evaluation site: Yes 


PROGRAM DETAILS 


Originally, the Social Innovation Fund (SIF) was a supplement to Workforce Innovation and 


Opportunity Act (WIOA) youth programming, with opportunity youth enrolled in both WIOA and SIF. 


This allowed partners to serve more opportunity youth while providing the same services they had 


through WIOA, which were informed by the Back on Track framework even before this grant. During 


the first year, youth received support from initial assessment and development through education or 


training, as well as help getting a job. Waiting lists were eliminated because the SIF funds enabled the 


partners to serve more youth. The backbone agency received guidance during this process from Jobs 


for the Future on how to deepen the Back on Track model, and further developments in 


implementation took place following the implementation study period. 
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OUTREACH AND RECRUITMENT METHODS 


Flyers 

Referrals from other community organizations 
Social media 

Outreach in local community (e.g., schools, malls) 


Word of mouth 


PRIMARY REASONS FOR ATTRITION 


Housing 

Food insecurity 
Child care 
Transportation 


Mental health 


PARTNERSHIPS 
Collective impact effort: Hartford Opportunity Youth Collaborative 


Backbone organization: Capital Workforce Partners 


Primary partners: 


Blue Hills Civic Association 


Our Piece of the Pie 
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New Orleans, Louisiana 

Back on Track approach: Postsecondary/career bridging 
Target population: Young people ages 16 to 24 

Number of young people served through August 2017: 314 


Impact evaluation site: No 


PROGRAM DETAILS 

The Cowen Institute at Tulane University in New Orleans supported four main programs through 
Opportunity Works: the Youth Empowerment Project (YEP), the largest literacy organization in the 
area; Delgado Community College’s Accelerated Career Education (ACE) program, based on the 
Integrated Basic Education and Skills Training (I- BEST) model; the Earn and Learn program, a paid 
work experience at Tulane University; and Bard Early College of New Orleans, which provided liberal 
arts college classes for youth. Opportunity Works allowed for increased capacity in these programs for 
opportunity youth who entered the system through any partner organization—a “no-wrong-door” 


approach. The programs provided case management and pathways to education and employment. 


Generally, the programs fell into the postsecondary bridging phase of the Back on Track model, 
but they were individualized to students’ needs. The Opportunity Works enhancements at YEP 
bridged to postsecondary education through services provided by transition coordinators. YEP 
participants also benefited from other YEP services, including High School Equivalency Test (HiSET) 
completion, work readiness, basic soft skills, employability skills, and help with opening a bank 
account, developing a budget, and obtaining official identification. Delgado’s ACE provided 
postsecondary bridging and industry-based credentials for those who wanted to pursue high-demand 
career and technical college courses. The program began with a noncredit industry-based 
credentialing semester; the subsequent semesters were for credit. The Opportunity Works 
enhancement helps the program concentrate resources toward opportunity youth. Earn and Learn 
provided an opportunity for paid work experience and on-the-job skills training linked to 
postsecondary education, mostly for those who have earned their high school diploma or high school 
equivalency (HSE). Opportunity Works allows Earn and Learn to serve more youth, doubling in 
capacity. In addition, Bard Early College taught an early college liberal arts writing and reading seminar 


at Earn and Learn and YEP. 


OUTREACH AND RECRUITMENT METHODS 


= Referrals from partner organizations 
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= Word of mouth 
= Information sessions 


= Advertising (e.g., radio, TV, social media) 


PRIMARY REASONS FOR ATTRITION 


= Housing 
= Transportation 
= Crime/safety 


= Mental health 


PARTNERSHIPS 
Collective impact effort: Employment and Mobility Pathways Linked for Opportunity Youth 


Backbone organization: Cowen Institute at Tulane University 
Primary partners: 


= Youth Empowerment Project 
= ACE Program at Delgado Community College 
= Earn and Learn Career Pathways Program 


= Bard Early College of New Orleans 


Philadelphia, Pennsylvania 
Back on track approach: Postsecondary/career bridging 


Target population: Young people ages 16 to 24 with or without high school credentials who are 
unemployed and/or not enrolled in postsecondary education, especially young men of color 


Number of young people served through August 2017: 120 


Impact evaluation site: Yes 
PROGRAM DETAILS 


The Philadelphia Youth Network (PYN) hired two instructors and two college/personal success 


coaches to work with enrollees in the Opportunity Works-sponsored College Success Program (CSP) 
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in the city’s four E3 high school reengagement centers. The CSP provided additional instruction and 
individualized support to about 80 enrollees a year. The staff members worked in coach-instructor 
pairs, and each team rotated between two E3 Centers. The CSP was divided into two phases. During 
the first 12-week phase, participants enrolled in one-hour college readiness courses four days a week 
with Opportunity Works instructors and coaches. The readiness course alternated daily between 
academic content (taught by instructors) and a noncognitive skills workshop (taught by coaches). 
Meanwhile, students engaged with E3 Center services, such as literacy support, high school 
equivalency instruction toward the General Education Diploma (GED) credential or a high school 
equivalency test, job readiness training, case management, and personal empowerment. During the 
second phase, the CSP staff helped participants bridge to postsecondary education by providing 
transition services, college placements, and ongoing support. Specifically, they helped with the 
logistics of dual or regular enrollment at the Community College of Philadelphia (CCP), accompanied 
participants to campus, monitored students’ experiences, and helped students plan for their next steps 
in postsecondary education and training. PYN covered the cost of a “first college experience,” based 
on the testing level and enrollment type of each student. The CSP staff continued to work with 
students after they enrolled in the Community College of Philadelphia for their first one to two 


semesters. 


OUTREACH AND RECRUITMENT METHODS 


= Building relationships with students at E3 Centers 


= Word of mouth 


PRIMARY REASONS FOR ATTRITION 


= Housing 

= Food insecurity 
= Child care 

= Transportation 
= Crime/safety 


= Mental health 


PARTNERSHIPS 


Collective impact effort: Project U-Turn 
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Backbone organization: Philadelphia Youth Network 
Primary partners: 


= Center for Literacy E3 Center 

= Communities in Schools of Philadelphia, Inc. E3 Center 
= Congreso de Latinos Unidos E3 Center 

= JEVS Human Services E3 Center 


= Community College of Philadelphia 


San Francisco, California 
Back on track approach: Enriched preparation 


Target population: African American and Latinx young people ages 17 to 24 without high school 
credentials 


Number of young people served through May 2017: 25 


Impact evaluation site: No 


PROGRAM DETAILS 

The Opportunity Works program in San Francisco worked with largely justice-involved Latinx and 
African American youth across the Bay Area who were not in school. The program offered enriched 
preparation activities to get them back in school, help them obtain their GED, move away from their 
past involvement with the justice system, and clear their records to achieve future success. The 
program supported these activities by working closely with an education partner, the Five Keys 
Charter School, to provide classroom activities and help students obtain their high school credential. 
They also worked with several other partners in the community to ensure that participants received 
supports to help them escape poverty and move past a justice-involved life. Five Keys also created a 
reentry plan to provide direct referrals to support services in the community. In addition, they 
coordinated with other service providers—including case managers, therapists, counselors, and the 
students/participants. This created a more successful outcome for youth while transitioning out of 


custody and reentering the community. 
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OUTREACH AND RECRUITMENT METHODS 
= Referrals from the San Francisco County Jail and Communities in Harmony Advocating for 


Learning and Kids (CHALK) 


PRIMARY REASONS FOR ATTRITION 


= Housing 


PARTNERSHIPS 


Collective impact effort: Roadmap to Peace 
Backbone organization: Bay Area Community Resources 
Primary partners: 


= Black to the Future 
= Communities in Harmony Advocating for Learning and Kids (CHALK) 


= Five Keys Charter School 


Santa Clara County, California 

Back on track approach: Enriched preparation 

Target population: Young people ages 16 to 24 without high school credentials 
Number of young people served through September 2017: 57 


Impact evaluation site: No 


PROGRAM DETAILS 

The Opportunity Works program in Santa Clara County consisted of three education navigators who 
served opportunity youth in the region, motivating them to complete their high school diploma and 
connecting them to needed support services. The three education navigators were embedded within 
local institutions and had a supervisor at each host site—Silicon Valley Community Foundation (SVCF), 
Fresh Lifelines for Youth (FLY), and Conxién to Community. These organizations serve pregnant and 
parenting youth (Conxi6dn), foster care youth (SVCF), justice-involved youth (FLY), and homeless 
youth—the four system touch points that rendered opportunity youth eligible for participation in the 
Opportunity Works program. Youth who have one of these system touch points triggered a referral to 


the Opportunity Works program when they enrolled in either San Jose Conservation Corps (SJCC) or 
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Opportunity Youth Academies (OYA), the two main educational service providers for dropouts in the 
region. A school staff member then connected them to the education navigator they felt could best 
serve the student’s needs. One education navigator focused on justice-informed youth, one on foster 
care youth, and one on justice-involved youth. Students with these touch points were typically 


referred to the corresponding navigator. 


The education navigator worked to motivate the youth to complete their education and provided 
referrals to other service providers to meet each student’s needs. The program’s goal was for students 
to receive a high school diploma, enroll in postsecondary education, and be excited about setting and 
working toward future goals. Opportunity Works students remained on their education navigator’s 
caseload while they pursued their high school diploma and for three months after beginning 


postsecondary education to provide a more seamless transition. 


OUTREACH AND RECRUITMENT METHODS 


= Referrals from education providers 


PRIMARY REASONS FOR ATTRITION 


= Housing 
= Child care 


= Transportation 


PARTNERSHIPS 
Collective impact effort: Opportunity Youth Partnership 


Backbone organization: Kids in Common 
Primary partners: 
= Opportunity Youth Academies of the Santa Clara County Office of Education 
= Conservation Corps 
= Fresh Lifelines for Youth 
= Silicon Valley Community Foundation 


=~ Conxidn to Community 
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South King County/Seattle, Washington 
Back on Track approach: Postsecondary/career bridging 


Target population: Young people ages 16 to 21 with or without high school credentials who are 
unemployed and/or not enrolled in postsecondary education, especially young men of color. 


Number of young people served through August 2017: 254 


Impact evaluation site: Yes 


PROGRAM DETAILS 

Seattle Education Access (SEA) partnered with Open Doors reengagement sites around South King 
County that provide high school completion supports through a mix of diploma and GED instruction, 
as well as case management to address basic needs such as stability, housing, and transportation. SEA 
added college prep navigators and retention services to supplement these existing services provided 
at Open Doors reengagement sites. While youth sometimes interacted in group settings, SEA’s 
intervention was mostly conducted one-on-one by talking about participants’ education histories and 
reasons for leaving traditional pathways and developing an individualized plan to successfully get to 
and through a college credential. SEA also worked with case managers at the Open Doors sites to 
make sure they had a coordinated plan to address any personal barriers to educational success that 
may exist for participants. The bulk of SEA’s intervention focused on career exploration, including 
one-on-one coaching; career assessments helping students identify areas of interest and career 
pathways; provision of labor market information; connection to alumni and people in the field; use of 
information to make decisions about a specific program; and support for financial planning, 


scholarships, and budgeting. 


OUTREACH AND RECRUITMENT METHODS 
= Partnerships with reengagement centers that assist youth in finishing high school or attaining 


a high school equivalency 
= Orientation meetings and classroom visits 


= Self-referrals 


PRIMARY REASONS FOR ATTRITION 


= Housing 


= Child care 
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PARTNERSHIPS 
Collective impact effort: Road Map Project 


Backbone organization: United Way King County (subgrantee) and Community Center for Education 


Results 
Primary partners: 


= Seattle Education Access (lead partner, working at the following Open Doors sites) 
= Youth Source Renton 

= YouthCare 

= jGrad Academy 

= Green River College Open Doors 

= Southwest Youth and Family Services 

= Career Link High School 


= Federal Way Open Doors 


Key Lessons from the Implementation Study 


The implementation study, conducted from the start of the Opportunity Works demonstrations in 
2015 through late 2016, was informed by document review, logic model mapping in coordination with 
the sites, quarterly calls, and site visits in fall 2016. The implementation study provided information 
about program fidelity and describes the sites’ efforts to inform others that may be interested in 
implementing similar interventions. The evaluation approach recognized the flexibility of Back on 


Track while ensuring that the programs bore some fidelity to Back on Track framework. 


The following sections list key takeaways from the implementation research. A fuller explanation 


of each implementation finding is available in Anderson et al. (2017). 


Nature of Communities 


= Overall, poverty rates across the seven communities ranged from 10 percent (Santa Clara 


County) to 33 percent (Hartford), with an average of 22 percent, compared with the US 
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average of 16 percent. Child poverty was particularly high in Hartford and New Orleans (45 


percent and 41 percent, respectively). 


= Every site reported that program participants are primarily people of color and come from 
communities experiencing significant economic hardship and instability. Common challenges 
included housing instability, access to services, lack of transportation, child care challenges, 


and exposure to crime and violence. 


Opportunity Works Programs 


= The seven sites in this study built their Opportunity Works programming from the Back on 
Track framework, adapting the framework to their local resources and context. All sites used 
all or nearly all of the framework elements and often went beyond the requirements of Back- 
on-Track phase on which they focused. They used the grant funds to fill in service gaps, 


primarily through hiring coaching and navigation staff. 


= In all sites, Opportunity Works brought some new or enhanced services, but typically features 
of the Back on Track framework already existed in the sites. In three sites, the new funds 


allowed more youth to be served. 


= Staff defined program success along five dimensions. Four were youth-focused dimensions, 
and one was systems focused. The youth-focused dimensions were (1) building 
personal/academic skills and aspirations, (2) developing the ability to live independently, (3) 
attaining educational benchmarks, and (4) embarking on a career pathway. At the systems 
level, the sites aspired to create a youth focus in the way that organizations and institutions 
developed and structured education and training programs and social services systems. The 
systems-level goals were partially supported by the Aspen Institute’s Opportunity Youth 


Incentive Fund collective impact effort for opportunity youth. 


= The concept of “disconnected” opportunity youth was not simple. The working definitions 
used by the sites for target population recruitment reflected a continuum of engagement 


around school, work, and career pathways. 


= The primary outreach and recruitment methods varied across sites, depending on the 
characteristics of the disconnected youth. Some sites were embedded within partner 
programs that enroll youth who have previously dropped out of high school, and the sites 


recruited directly from that program’s student population. 
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In all sites, the bulk of funds supported staff positions, primarily to provide or enhance 
education-related support services and improve organizations’ ability to help youth navigate 
support and education systems. In some sites, there was an intentional effort to hire staff 


whose backgrounds make them more relatable to the youth. 


Sites report that limited funds required them to make a trade-off between funding staff at 
sufficient levels to individualize supports or directly supporting services that meet the basic 
needs of youth; they rely primarily on community resources for supportive services. Most 
communities were unable to meet the basic needs of youth in housing, child care, 
transportation, food security, crime/safety, and mental health; these are the reasons that 


youth tend to leave before finishing the program. 


Sites used multiple strategies to improve youth retention, including frequent contact with 
youth, relationship building, and monetary incentives. Three of the seven sites also articulated 


“recovery” strategies to reenroll youth who drop out of the program. 


Although the education-related services were the most clearly articulated part of the Back on 
Track framework, the fact that sites incorporated elements and features of the framework 
across phases suggests the phases of the Back on Track framework may be inseparable. JFF 
took steps to address this through a revised program model, released in draft form in late 


2016. 


Lessons for Others Interested in Developing Similar Programming 


DESIGN 


When first thinking about program design, various sites emphasized the importance of having 
a strong understanding of the local labor market and the population to be served. One 
valuable way to get this insight is to engage program partners at multiple levels and possibly 


young people themselves to provide feedback. 


Many sites found recruitment to be an unexpected challenge. The difficulty was often in 
finding target youth who were ready for meaningful program engagement, especially intensive 
postsecondary bridging experiences. Anticipating and planning for recruitment and retention 


challenges can save the energy of shifting focus partway through implementation. 


Sites cautioned that it is important to ensure that program components are logistically 


coordinated and that young people are supported in accessing them. Academic institutions 
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often have inflexible procedures and schedules. It is good to recognize and try to work 


through the alignment of program elements early in planning. 


Offering nonacademic content, such as opportunities for socialization and cultural capital 
building, college- and job-readiness skill development, and personal confidence building can 


complement standard programming. 


Ensuring that new interventions are embedded seamlessly into existing programming can 
offer an uninterrupted continuum for youth that may increase retention and improve 


outcomes. 


Strong technical support from the funder or another organization well-versed in the 


framework can be valuable at the planning stage and throughout implementation. 


SELECTING AND WORKING WITH PARTNERS 


DATA 


Backbone organizations cautioned that it is important to select partners carefully. It is 
valuable to have partners with a range of service offerings and strengths to address the young 


people’s various needs. 
It is also helpful to define the roles and expectations of partners clearly from the outset. 


When new partners come to the table, establishing trust is often a necessary first step, but it 


can take time. 


Regular partner meetings organized by backbone organizations can give partners a chance to 
create a common language, vision about the program, and data/tracking procedures. These 
meetings may be particularly effective when they allow for interaction among leadership, 


management, and direct service staff on multiple levels. 


It may be necessary to offer training to partners on the program goals, data collection and 


usage, and other key issues or skills. 


It is valuable to build a data culture, which is an iterative and interactive process. Partners 
need to capture useful and usable information that can inform programming, not just meet 
reporting requirements. Shared data can allow staff across organizations to build a “conspiracy 


of support” for participants. 
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STAFFING AND WORKING WITH YOUTH 


The Back on Track framework is personnel intensive, and hiring the right staff members is a 
critical ingredient for success. Successful staff were relatable, set and maintained high 
expectations for youth, and demonstrated that they cared and understood young people’s 


backgrounds. 


It was valuable for staff to have an opportunity to build trusting relationships with the youth 


as part of the program design. Establishing relationships took time and attention. 


Ensuring a reasonable staff-to-student ratio may maintain quality services and minimize the 


chance of staff burnout. 


Young people’s barriers loomed large, especially around housing, transportation, debt, the 
need for income, hunger, mental health, transportation, and unsupportive social networks or 


family members. 


Framework 


It was difficult to have meaningful postsecondary bridging without strong enriched 
preparation elements. Conversely, it was hard to promote an attractive enriched preparation 
program without identifying postsecondary bridging opportunities as a next step. In the 
original Back on Track framework, the phases were inherently intertwined. In response to this 


lesson, the revised framework incorporates redundancies in each phase. 


The framework may have needed to incorporate some earning opportunities for youth. The 


pressing need for income is a fundamental barrier that affects program progress and success. 


Participant Perspectives from Focus Groups 


Youth perspectives highlighted some of the most salient lessons from the implementation research 


and provided valuable feedback for future programming. The young people who attended the focus 


groups shared that they and others in the program came from difficult backgrounds and discussed 


their history and current challenges with the justice system, transportation, pregnancy and parenting, 


personal violence, housing/homelessness, illegal activities, and mental illness. 


Some youth were initially unclear about their goals, and the program helped them see a path or a 


purpose. Others wanted to go to college all along but did not know how. Many had to balance the 
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need for a job with the desire to advance their education so that they could pursue a range of 
professional, long-term interests. Youth in programs with a financial incentive expressed that this 
feature was important to help them stay engaged in the programming. Those who were in programs 


without stipends or paid work experience opportunities wished there was more financial support. 


Overall, most expressed positive experiences with the program. Almost universally, enrollees said 
that it helped them find direction or explore avenues they had not previously considered, such as 
college offerings aligned with their substantive interests. Many expressed pride and satisfaction about 


their accomplishments in the program. 


Youth expressed the importance of relatable and caring staff, being held to high expectations and 
having peer support networks, a voice in program design, and a safe place with sufficient space. They 
valued the support services offered, but some wanted additional supports, such as transportation, 


child care, more income, help with time management, and mental health support. 


Insights from the Program Data 


Each Opportunity Works site collected their own administrative program data to track participant 
characteristics and activities within Opportunity Works. Information on program activities across the 
seven sites for all enrollees ages 18 and older illustrate the nature and characteristics of Opportunity 
Works participants and programming. Because the program data do not track participant outcomes in 
a consistent way, have a limited number of participant well-being indicators, and do not contain 
information about comparison group members, these data cannot help describe program outcomes or 


effects. 


The research team worked with sites to improve their data quality and gave general guidance 
about the types of indicators that would be useful to track. However, because of differences in 
tracking systems, variable definitions, and programming populations and approaches, most data 
elements are not directly comparable across sites. The following summaries are largely reported ona 
site-specific basis, except where it is reasonable to place sites next to each other to contextualize the 
results. The Urban Institute has not audited these numbers. In most cases, they reflect the latest 


available data as of summer 2017, though the figures are cumulative. 
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Participant Demographics 

The following graphs describe participant characteristics as documented in the program data. Because 
each site did not track every characteristic in the same way, many tabulations are limited to a subset 
of sites. Participants with missing data do not contribute to the percentages, and the sample size for 


each tabulation in each site appears on the horizontal axes of the graphs. 


A slight majority of program participants were male (52 percent), with San Francisco and Hartford 
serving a higher proportion of male participants than the other sites. Overall, 46 percent of 
participants were young men of color, defined as non-white male participants. This is expected, given 


the explicit focus on young men of color in the Opportunity Works initiative. 
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Few participants identified as non-Hispanic white. In four sites, most participants identified as 


non-Hispanic Black. In the two California sites, most participants identified as Latinx. South King 


County/Seattle had the most diverse mix of participants. 


Participants averaged around 20-years-old at intake, though New Orleans participants were 


slightly younger on average. Boston, Philadelphia, and South King County/Seattle served some 


participants who were over age 24 at intake. 
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FIGURE 8 
Race/Ethnicity 
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FIGURE 9 
Age at Intake 
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Participants’ educational preparation level, as recorded in the program data, varied widely among 
sites. In Philadelphia, almost no participants had a high school credential at enrollment (96 percent had 
less than a high school credential), while in Hartford almost all enrollees had a high school credential 
when they entered the program. Hartford also had the most enrollees—over one in five (21 percent)— 
who reported some college experience when they entered Opportunity Works. These variations 


reflect differing target populations among the sites. 
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FIGURE 10B 
FIGURE 10A Some College Experience Reported 
Less than a High School Credential 
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As expected, the young people enrolled in the Opportunity Works programs faced a range of 
challenges, including having children (23 percent), previous or current foster care involvement (10 
percent), ever or currently homeless (20 percent), or ever justice-system involved (30 percent). The 
young people in Santa Clara were particularly likely to have current or previous involvement in the 
foster care and/or justice systems and to be pregnant or parenting. Young people in South King 


County/Seattle were more likely to have experienced homelessness than participants in other sites. 
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FIGURE 11B 
ESURE SEA . Ever or Currently in Foster Care 
Pregnant or Has Children 
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FIGURE 11C 
Ever or Currently Homeless FIGURE 11D 
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Program Activities 


The program activities data provide some insight on the types of programming in which participants 
engaged and the intensity of their involvement. These data are less consistent across sites than the 


participant characteristic data, so tabulations for each site appear separately. 
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BOSTON, MASSACHUSETTS 

In Boston, many participants were still enrolled as of the latest available data. However, the site 
struggled to retain young people in the college/career bridging program (called “College Bridging”), 
with an attrition rate of 27 percent. Training program completion was much higher, though the 
withdrawal rate was only somewhat lower. The higher completion rate in training programs relative to 
the college bridging programs likely occurred because training programs were shorter than college 


academic programs. 


FIGURE12A 
College Bridging Status (N = 41) 
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FIGURE 12B 


Training Status (N = 42) 
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HARTFORD, CONNECTICUT 

In Hartford, most participants for whom data were available engaged in the program for 100-199 
days, but about 13 percent engaged for over 300 days." All participants completed a standardized 
skills assessment, and nearly all received guidance and counseling (96 percent). A very high portion 
also set goals and completed occupational skills training (89 percent each). Over half (52 percent) 


received job development and supports. 


FIGURE 13 FIGURE 14 
Days in the Program (N = 62) Most-Completed Program Activities 
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NEW ORLEANS, LOUISIANA 

In New Orleans, participants were divided nearly equally between programming at Delgado 
Community College (48 percent of all participants) and the Earn and Learn Career Pathways Program 
(52 percent of all participants). Participants at Delgado Community College were enrolled relatively 
evenly across its three programs: Core Programming, Education and Employment, and Support and 
Stabilization, whereas most participants enrolled at Earn and Learn were involved in Core 


Programming and a much smaller percentage were involved in Education and Employment. # 


Across all participants in the city, four in ten (41 percent) completed a standardized skills 


assessment and 10 percent set goals. 
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FIGURE 16 


FIGURE 15 Benchmarks 
Program Participation (N = 83)18 
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PHILADELPHIA, PENNSYLVANIA 

Nearly three-quarters (73 percent) of participants in Philadelphia attended a writing skills class, and 
almost as many participated in classes or workshops on understanding college expectations, goal 
setting, and college and career planning. Just over half (52 percent) participated in self-motivation 
classes. Participants engaged in a range of sessions focused on issues such as well-being and college 
planning. Group meetings ranged from one to two hours, with most sessions closer to two hours. One- 
on-one meetings could also last two hours, but the average length was just over half an hour. Staff 


also interacted with participants through a range of other media. 


The most common reason for dismissal among the 62 participants with data on dismissal or 
completion was that participants resigned from the program (34 percent). Only 19 percent had 


completed their goals. Some less common reasons were leaving for employment or low attendance. 
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FIGURE 17 
Most-Attended Classes or Workshops 
(N = 120) 
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SAN FRANCISCO, CALIFORNIA 

About two-thirds (64 percent) of participants in San Francisco were dually enrolled in Opportunity 
Works and Roadmap to Peace, while the remainder were only in Opportunity Works. Participants 
engaged in a range of activities, including Step to College, completing an individual plan or career 


action plan, and/or taking a GAIN basic skills assessment. 
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FIGURE 20 
Program Enrollment (N = 25) 
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SANTA CLARA, CALIFORNIA 


FIGURE 21 
Activity Participation (N = 25) 
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Youth in Santa Clara enrolled in one of three partner sites, with San Jose Conservation Corps as the 


most common enrollment site (48 percent). About half of youth had started or completed a 


postsecondary plan as of the latest data collection. Staff were in intensive contact with some youth, 


with 12 contacts and 18 meetings for the median young person, but only 39 percent of young people 


were considered in “regular contact,” ‘4 and nearly one-quarter (23 percent) had little or no contact 


with program staff. 1 


FIGURE 22 
School Program (N = 42) 
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FIGURE 23 
Postsecondary Plan (N = 57) 
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FIGURE 24 


Median Number of Contacts between Staff 


and Youth (N = 57) 


Contacts per youth Meetings per youth 


SOUTH KING COUNTY/SEATTLE, WASHINGTON 
South King County/Seattle’s Opportunity Works program participants were spread across many 
partner sites, but the most common sites were Career Link and iGrad. As of the latest available data, 


only 7 percent of participants were inactive, while almost one-third (31 percent) had graduated from 


FIGURE 25 
Engagement Level (N = 57) 
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the program” and the remainder were still active in the program (62 percent). 
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FIGURE 27 


FIGURE 26 Program Status (N = 252) 
Enrollment Site (N = 252) 
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Well-Being Indicators 


Only a subset of sites provided data on participant well-being after program enrollment. A challenge in 
reporting well-being or participant outcomes based on program data is that programs often only track 
participants who stay in touch. Those who stayed in contact with program staff tended to be more 
stable and/or had better program experiences. Therefore, these findings are only descriptive and 
should be interpreted with appropriate caution. The impact section of this report provides objective 


outcomes and effects for three program sites. 


BOSTON, MASSACHUSETTS 
Boston reported few outcomes beyond credential attainment, but the latest data showed that one- 
quarter of participants (25 percent) had earned a certificate. Few had earned a higher-level credential 


(e.g., a bachelor’s or associate’s degree), though more might emerge with additional follow-up. 


| OPPORTUNITY WORKS FINAL REPORT 


FIGURE 27 
Credentials Earned (N = 182) 
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NEW ORLEANS, LOUISIANA 

In New Orleans, 1 in 10 participants completed a college application. Of those who attended a 
postsecondary institution, over half (58 percent) attended a two-year college, nearly one-quarter (22 
percent) attended a four-year college, and the remainder (20 percent) were in a dual enrollment 
program. Of the students who entered postsecondary and had information, 14 percent had completed 


the first year of their postsecondary placement at the time of the latest available data. 1” 


FIGURE 28A 
College Application (N = 230) FIGURE 28B 
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FIGURE 29 
Type of Institution Attended (N = 50) 
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SAN FRANCISCO, CALIFORNIA 

In San Francisco, nearly three-quarters (72 percent) of participants had a resume, over half (56 
percent) had a bank account, and many had a cover letter or had made progress toward their driver’s 
license by the latest reporting period. On a scale of one to five, participants rated that they had the 
highest confidence in understanding financial outcomes but were least confident in their familiarity 


with financial aid. 


FIGURE 30 FIGURE 31 
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SANTA CLARA, CALIFORNIA 

Santa Clara carefully tracked participants’ self-sufficiency growth over time; though the amount of 
time that participants had been in the program between intake and the most recent measure may vary 
across participation based on when they enrolled, the records do not note the timing of the most 
recent measure. Based on these simple descriptive growth measures, participants saw progress in 
income security, educational attainment, food security, and health care by the most recent data 
collection. They did not see progress on housing stability, transportation, or legal issues. These figures 
should not be interpreted as representing program effects because many people were still enrolled in 
programming as of the most recent measure and because the figures are not aligned with a meaningful 


comparison. 


FIGURE 32A 
Self-Sufficiency: Income FIGURE 32B 
Self-Sufficiency: Housing 
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FIGURE 32C 
Self-Sufficiency: Education 
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FIGURE 32E 
Self-Sufficiency: Transportation 
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FIGURE 32D 
Self-Sufficiency: Food 
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FIGURE 32F 
Self-Sufficiency: Legal 
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FIGURE 32G 
Self-Sufficiency: Health Care 


mw At intake (N = 49) mMost recent (N = 54) 


None 


Medical coverage 


9 
Medical coverage and are 


medical home 
65% 


SOUTH KING COUNTY, WASHINGTON 

In South King County/Seattle, the most common outcome was college exploration or placement, 
which accounted for 14 percent of all outcomes. Eleven percent achieved a high school equivalency, ‘8 
and 10 percent completed a college assessment or were accepted to college. Among those who 
attended a degree program, participants were most likely to be enrolled in a two-year transfer 


postsecondary program (64 percent). 
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FIGURE 34 
FIGURE 33 Degree Program (N = 96) 
Top Student Outcomes (N = 980 outcomes; 
students may appear more than once) 


Four-year university 
College 
exploration/placement 14% 


initiated 
Two-year transfer 64% 


GED or high school 


9 

diploma earned 11% 
College assessment Technical yeat* 

10% 

completed 
Technical—short-t 

Accepted to college 10% a 
Financial aid paperwork 3% Apprenticeship 


completed 


Program Data Summary 


The program data are not always comparable, but they demonstrate the types of measures that the 
Opportunity Works sites deemed important to track. They also provide some additional context about 


participant characteristics, programming details, and suggestive indicators of well-being. 


These diverse data point to the need for a consistent data collection effort for cross-site 
evaluation, which is why the research team developed the baseline and follow-up participant surveys. 


Some descriptions of program experiences from the survey data appear in the next section. 


Descriptions of Program Activities from the Follow-Up 
Survey 


Though the baseline survey was offered to participants at all seven sites, budget constraints and 
evaluation needs made it so only participants at impact sites could receive cash incentives for 
participation. This severely limited responses at the four nonimpact Opportunity Works sites. 


Ultimately, only participants in Hartford, Philadelphia, and South King County/Seattle—as well as 
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comparison group members at each of those sites—completed the follow-up survey. This section 


describes some of their program experiences. 


The follow-up survey asked respondents to describe the types of services they received using 
three categories: education, job or career, and other. Education-related services included enrolling in a 
college class or completing their high school diploma or equivalency. Job- or career-related services 
supported students by providing technical training for a career or preparing them for short- or long- 
term employment. Finally, services in the “other” category included opportunities for one-on-one 
coaching and referrals to other service providers. Some of the tabulations below aggregate among 


respondents across these three impact study sites, while others provide site-specific information. 


Those in the treatment group answered two separate questions about services provided through 
Opportunity Works and services they received through other programs. Figure 35 illustrates the top 
five types of services that students received through Opportunity Works programming, most of which 


fell into the education and job-related service categories. 


FIGURE 35 
Most Students in the Treatment Group Received a Variety of Education- and Career-Related 
Services through Opportunity Works 


= Opportunity Works m= Other program 


Learn about different colleges (education) 
Information about college success (education) 
Enrolling in a college class (education) 

Setting short-term or long-term career goals (career) 


Job-ready skills (career) 


0% 20% 40% 60% 80% 100% 


Share of respondents who received the service 


Source: Opportunity Works Social Innovation Fund follow-up survey. 
Note: The denominator does not include respondents who indicated “Prefer not to answer” in their survey. 


Eighty-eight percent of students learned about different colleges and what it would take to be 
successful in that environment. At least 8 in 10 students received help enrolling in a college class, 


setting short-term or long-term career goals, and developing job-ready skills. 


Student experiences varied by site. For example, one-on-one coaching ranked first among the 


respondents in the Hartford treatment group, reported by over 90 percent of students; transportation 
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assistance ranked highly at 84 percent. Similarly, 89 percent of students in the South King County 
treatment group indicated that they had received help completing financial aid. The Philadelphia 
treatment group reported relatively higher shares of respondents receiving education- and career- 
related services compared with the other two sites, with South King County/Seattle respondents 
reporting lower exposure to career-related services (figure 36a and figure 36b). Students in the 
Hartford treatment group tended to receive additional supports not directly related to education or 
employment, such as referrals and one-on-one coaching, at higher rates compared with the other two 


sites (figure 36c). 


FIGURE 36A 
Philadelphia Treatment Group Reported Somewhat Higher Shares of Respondents Receiving 
Education-Related Services 


@Hartford Philadelphia mSouth King County 


93% 85% 85% 


74% 73% 


Complete high Math, reading, Learn about Enrolling in a Complete Information Field trips 
school or writing skills different colleges college class financial aid about college 
equivalency success 


Source: Opportunity Works Social Innovation Fund follow-up survey. 
Note: The denominator does not include respondents who indicated “Prefer not to answer” in their survey. 
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FIGURE 36B 


Philadelphia and Hartford Treatment Groups Reported Somewhat Higher Shares of Respondents 
Receiving Career-Related Services 


@Hartford Philadelphia mSouth King County 


87% as i 
‘0 


81% 


76% 


76% 


59% 60% 58% 


Career options Setting short-term Technical training Short-term Long-term Job-ready skills 
or long-term career for a career employment employment 
goals 


Source: Opportunity Works Social Innovation Fund follow-up survey. 
Note: The denominator does not include respondents who indicated “Prefer not to answer” in their survey. 


FIGURE 36C 
Hartford Treatment Group Reported Relatively Higher Shares of Respondents Receiving Additional 
Supports Not Directly Related to Education or Employment 


@Hartford Philadelphia mm South King County 


91% 
83% 83% 84% 85% 


32% 31% 


One-on-one coaching Supportive Referrals to housing, Help with getting Transportation help 
environment with peers child care, counseling, driver's license 
etc. 


Source: Opportunity Works Social Innovation Fund follow-up survey 
Note: The denominator does not include respondents who indicated “Prefer not to answer” in their survey. 
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In contrast, students in the comparison group ranked services received differently; most of these 
differences between the treatment and comparison groups overall were statistically significant at the 
0.01 level. The only services where there was no statistically significant difference between the 
groups were completing high school equivalency; referrals to housing, child care, or counseling; and 


help getting a driver’s license (figure 37 through figure 39). 


FIGURE 37 
Higher Shares of Respondents in the Treatment Group Received Education-Related Services Than 
Those in the Comparison Group 


= Treatment = Comparison 


89% 88% 89% 


Complete high Math, reading, Learn about Enrolling in a Complete Information Field trips*** 
school or writing skills*** different college class*** financial aid*** about college 
equivalency colleges*** success*** 


Source: Opportunity Works Social Innovation Fund follow-up survey. 
Notes: The denominator does not include respondents who indicated “Prefer not to answer’ in their survey. Services marked 
with *** indicate that treatment and comparison group responses were statistically significant at the 0.01 level (p < 0.01). 
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FIGURE 38 


Higher Shares of Respondents in the Treatment Group Received Career-Related Services Than Those 
in the Comparison Group 


& Treatment = Comparison 


86% 


Career options*** Setting short-term Technical training Short-term Long-term Job-ready skills*** 
or long-term career for a career*** employment*** employment*** 
goals*** 


Source: Opportunity Works Social Innovation Fund follow-up survey. 
Notes: The denominator does not include respondents who indicated “Prefer not to answer’ in their survey. Services marked 
with *** indicate that treatment and comparison group responses were statistically significant at the 0.01 level (p < 0.01). 


FIGURE 39 


Higher Shares of Respondents in the Treatment Group Received Other Services Not Directly Related 
to Education or Employment 


= Treatment = Comparison 


81% 81% 77% 


28% 29% 


One-on-one Supportive environment Referrals to housing, Help with getting Transportation help*** 
coaching*** with peers*** child care, counseling, driver's license 
etc. 


Source: Opportunity Works Social Innovation Fund follow-up survey 
Notes: The denominator does not include respondents who indicated “Prefer not to answer” in their survey. Services marked 
with *** indicate that treatment and comparison group responses were statistically significant at the 0.01 level (p < 0.01) 
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As shown in figure 40, two of the five top services respondents in the comparison group received 
were in the “other” category: a supportive environment among peers and transportation assistance 
(e.g., free or subsidized bus passes). This contrasts with the top five services reported for the 


treatment group in figure 35. 


FIGURE 40 
About Two-Thirds of Students in the Comparison Group Received a Support Beyond Education- and 


Career-Related Services 


Supportive environment with peers (other) 67% 
Job-ready skills (career) 64% 
Learn about different colleges (education) 62% 
Setting a career goals 60% 
Transportation help (other) 57% 


Source: Opportunity Works Social Innovation Fund follow-up survey. 
Note: The denominator does not include respondents who indicated “Prefer not to answer” in their survey. 


Again, the share of students who received specific services varied by site. For example, in South 
King County/Seattle and Philadelphia, about two-thirds of respondents had opportunities to complete 
high school or an equivalent degree, whereas only about a quarter of respondents in the Hartford 
comparison group said the same. This is likely because almost all students in Hartford had a high 


school credential at program entry. 


Overall, a greater share of respondents from the treatment group indicated that they had received 
any kind of service across all three categories relative to those in the comparison group (figure 41). 
This demonstrates that there were important differentials in program experiences between members 
of the treatment and comparison groups, though many members of the comparison group received 


some level of service. 
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FIGURE 41 

A Larger Share of Respondents from the Treatment Group Indicated That They Received Any 
Services Related to Education, Employment, or Additional Supports, Relative to Those in the 
Comparison Group 


= Treatment = Comparison 


97% 


RK RK 


Any education-related service Any employment-related service*** Any other service 


Source: Opportunity Works Social Innovation Fund follow-up survey. 
Note: The denominator does not include respondents who indicated “Prefer not to answer” in their survey. Services marked 
with *** indicate that treatment and comparison group responses were statistically significant at the 0.01 level (p < 0.01). 


Impact Study Implications 


While members of the comparison group participated in some programming, consistent treatment 
differentials appear between the groups. That is, Opportunity Works participants who responded to 
the follow-up survey received more intensive services than comparison group members enrolled in 


“business as usual” programming. 


Despite the observed differences in service quantity, the survey data do not distinguish “more” 
from “better” services. For example, the survey does not characterize the quality of the high school 
equivalency programming, college guidance, or other supports. However, the implementation study 
and program data suggest that the Opportunity Works grant effort caused all providers, even those 
who were continuing a version of services that existed before Opportunity Works, to carefully 
consider the nature of their intervention, develop better ways to use data to track participation and 
outcomes, and bring partners to the table in more meaningful and sometimes new ways to create 


better interventions for youth. 


PROGRAM OPERATIONS 45 


An important note is that the nature of programming varied widely among the three impact sites, 
though all of them focused on postsecondary bridging. Philadelphia had the strongest focus on 
education interventions, though South King County also had a strong focus on education-related 
services. Hartford was strongest in the “other” services category. Regardless of these differences in 
focus, each site had elements of all three that largely aligned with the Back on Track model.” This 
variation means that the primary common element across sites was the guidance of the Back on Track 
framework. Consistency across the sites would provide further evidence that Back on Track’s 
postsecondary bridging intervention itself should be credited with the positive effects, rather than the 


effect of any specific intervention approach. 
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Impact Study Methods 


This chapter describes the technical methodology employed in the Opportunity Works impact study. 
The impact study employs a technique called “propensity score matching” to find members of a 
comparison group who are most like treatment group members on observed characteristics. This 


quasi-experimental approach meets the Social Innovation Fund’s definition of “moderate evidence.” 


Impact Study Research Questions and Outcomes 


At the beginning of this project, the research team defined seven research questions that would be 
the focus of the impact study, three being confirmatory and four exploratory. The differentiation 
between confirmatory and exploratory questions is valuable to help policymakers focus on the 
strongest evidence of whether a program worked while also examining other desirable effects that 


might have been generated or effects for certain subgroups. 


The original evaluation plan was premised on a five-year study period. However, federal cuts to 
SIF grant funds limited the evaluation period to only three years. Therefore, the research questions 
have been modified to reflect outcomes that the research team could measure in the observed time 


frame. 


Confirmatory Questions 


1. What is the effect of the Back on Track program on participants’ education and career 


outcomes? 


2. Are participants in the program more likely to2° 


receive a high school diploma or GED? 
apply to a training program? 

apply to college? 

enroll in a training program? 

e. enroll in college? 


aoo0» 


3. After completion of the program, are participants in the program? 


a. less likely to be disconnected from work and education? 
b. less likely to be looking for work (if they are not pursuing education)? 22 


Exploratory Questions 


4. Are participants in the program likely to have fewer subsequent arrests or incarceration 


spells? 


5. Are participants in the program likely to have a stronger care network? 


6. Dosome sites have larger effects than others? 


7. Howare program effects different by subgroups? 


a. Is the effect of the program different for young men of color?7° 


Outcomes 


Given these research questions, the research team estimated the effects of the outcomes listed in 


table 1. Subsequent discussion describes the data sources. Outcomes from the follow-up survey are 


observed only for a share of the sample because of survey nonresponse. 


TABLE 1 


Outcome Measures for the Opportunity Works Impact Evaluation 


Outcome measure 
Enrolled in any 
postsecondary 
institution 


Enrolled in a two-year 
postsecondary 
institution 


Enrolled in a four-year 


postsecondary 
institution 


High school credential 


Applied to an associate’s 
degree program 


Data 
source 
NSC 


NSC 


NSC 


Survey 


Survey 


Definition 
Matched with the National Student Clearinghouse 
(NSC) as having an enrollment date for any 
postsecondary institution following the date the 
student completed the baseline Opportunity Works 
survey. 
Matched with the NSC as having an enrollment date 
for any institution that the NSC coded as “2-year” 
following the date the student completed the 
baseline Opportunity Works survey. 
Matched with the NSC as having an enrollment date 
for any institution that the NSC coded as “4-year” 
following the date the student completed the 
baseline Opportunity Works survey. This may include 
some community colleges that offer four-year degree 
programs. 
Answered “yes” to survey question “Have you 
received a high school diploma?” or “Have you 
received a GED, HiSET, or TASC high school 
equivalency?” 
Chose “An associate’s degree program at a two-year 
or community college” (as one of multiple potential 
responses) in response to the question “Since 
[baseline survey date], did you apply to a degree or 
credential program at any of the following?” 


Percent of 
study sample 
with measure 
96% 


96% 


96% 


61% 


60% 
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Data 
Outcome measure source 
Applied to a four-year Survey 
degree program 


Applied to a training Survey 
program 
Completed FAFSA Survey 


Attended an associate’s Survey 
degree program 


Attended a four-year Survey 
degree program 


Attended any Survey 
postsecondary 

institution 

Attended a training Survey 
program 

Looking for a job Survey 
Not working and not Survey 


enrolled in school 
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Percent of 
study sample 
Definition with measure 
Chose “A four-year degree program at a college or 60% 
university” (as one of multiple potential responses) in 
response to the question “Since [baseline survey 
date], did you apply to a degree or credential program 
at any of the following?” 
Chose “A training program focused on a specific job, 60% 
industry, or occupation” (as one of multiple potential 
responses) in response to the question “Since 
[baseline survey date], did you apply to a degree or 
credential program at any of the following?” 
Answered “yes” to survey question “Did you 56% 
complete the Free Application for Federal Student 
Aid (FAFSA) or another form requesting financial aid 
for college?” 
Chose “An associate’s degree program at a two-year 60% 
or community college” (as one of multiple potential 
responses) in response to the question “Since 
[baseline survey date], did you attend a degree or 
credential program at any of the following?” 
Chose “A four-year degree program at a college or 60% 
university” (as one of multiple potential responses) in 
response to the question “Since [baseline survey 
date], did you attend a degree or credential program 
at any of the following?” 
Chose “An associate’s degree program at a two-year 60% 
or community college” and/or “A four-year degree 
program at a college or university” (as one of multiple 
potential responses) in response to the question 
“Since [baseline survey date], did you attend a degree 
or credential program at any of the following?” 
Chose “A training program focused on a specific job, 60% 
industry, or occupation” (as one of multiple potential 
responses) in response to the question “Since 
[baseline survey date], did you attend a degree or 
credential program at any of the following?” 
Answered “no” to survey question “Last week, did 42% 
you do any work for pay?” and answered “Interested 
in working but can't find work” to survey question 
“What is the main reason you are not currently 
working?”—only for respondents not enrolled in 
school. 
Answered “no” to survey questions “Are you still 56% 
attending this college?” and “Are you still in this 
training program?” (for those who indicated they 
enrolled), and answered “no” to survey question “Last 
week, did you do any work for pay? If you had a job 
but did not work last week because of a vacation or 
illness, please count that as working.” 


Percent of 


Data study sample 
Outcome measure source Definition with measure 
Goal to graduate from a Survey Chose “Graduate from a two-year college program” 62% 
two-year program (as one of multiple potential responses) in response to 


the question “Suppose you could do just what you'd 

like and nothing stood in your way. How many of the 

following things would you want to do?” 
Goal to graduate from a Survey Chose “Graduate from college (four-year program)” 62% 
four-year program (as one of multiple potential responses) in response to 

the question “Suppose you could do just what you'd 

like and nothing stood in your way. How many of the 

following things would you want to do?” 


Arrested since baseline Survey Answered “yes” to survey question “Have you been 55% 
arrested since [baseline survey date]?” 
Social Provision Score Survey Seven-item social provision scale 54% 


Notes: “NSC” stands for National Student Clearinghouse. “Survey” references the Opportunity Works follow-up survey. The 
study sample is defined by program participants who took the baseline survey. The percent of study sample with the measure 
illustrates the outcome nonresponse. 


Comparison Groups 


In consultation with the Opportunity Works program staff, the Urban team determined appropriate 
comparison programs from which to recruit youth by selecting programs in the same region as the 
Opportunity Works sites serving an underemployed, out-of-school youth population but offering 
services distinct from the Back on Track framework. The advantage of recruiting from other non- 
Back-on-Track programs is twofold: first, it shows how the Back on Track model fares in comparison 
with “business as usual” youth programming. Second, it compares Opportunity Works participants 
with members of the comparison group who were similarly motivated to join a program to improve 
their education and/or career outcomes. This motivation and desire to engage in programming is an 
important selection element that can be difficult to measure if not observed empirically through 


program enrollment. 


Hartford 


Comparison group members in Hartford were recruited through a multipronged effort. Our Piece of 
the Pie (OPP) and the Blue Hills Civic Association (BHCA) were running other less intensive programs 
to support youth in forging career pathways. When youth enrolled in these programs, Hartford Youth 
Service Corps at OPP and Career Pathways for Youth at BHCA, they were invited to take the 


comparison group survey. The Urban League of Greater Hartford also recruited eligible youth who 
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sought services from them. These three programs explained the study and recruited youth in person. 
The final group, Hartford’s American Job Center, contacted youth who previously sought services 
from them by phone or provided Urban Institute staff with participant information for them to contact 
by phone. All youth recruited from each comparison program were screened to be sure they were not 
participants in the treatment group programs and that they had a high school diploma or GED and 
were ages 18 to 24. Screening comparison group members in Hartford differed from the other two 


sites, where having a high school credential was not a criterion for program enrollment. 


Philadelphia 


The primary source of comparison group members in Philadelphia was students enrolled in the School 
District of Philadelphia’s accelerated high schools. Students in these schools did not succeed in 
traditional high schools. They were over age and undercredited, but they were pursuing an alternative 
high school credential in a programmatic setting. They did not have the opportunity to participate in 
Opportunity Works programming because they were not affiliated with the Opportunity Works 
program sites. The research team was able to recruit students from two accelerated high schools with 
the cooperation and permission of the school district. One difference between these students and 
those enrolled in Opportunity Works is that students in accelerated high schools were pursuing an 
alternative high school diploma, while Opportunity Works students in the city’s E3 Centers were 


pursuing a GED or high school equivalency. 


South King County/Seattle 


Students from other Open Doors reengagement sites outside South King County, who are not served 
by the Back on Track program, were in the comparison group. The primary comparison recruitment 
location was the Goodwill Open Doors reengagement site in Tacoma, WA. These students were 
demographically and socially similar to South King County participants who were offered the 
opportunity to benefit from Opportunity Works services. One difference between the Tacoma 
comparison group students and those enrolled in Opportunity Works is that students in the Tacoma 
program were pursuing GED credentials, while Opportunity Works students in the Opportunity Works 


Open Doors center were pursuing a mix of GEDs and diplomas. 
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Data Collection Methods 


Participant Baseline and Follow-Up Surveys 


Opportunity Works enrollees in impact sites were invited and incentivized to take a baseline and 12- 
to-17-month follow-up survey. The surveys provide a useful mechanism for data collection because 
they cover not only program participants but also people from identified comparison programs, as 


discussed previously. 


The baseline survey was designed to capture participants’ characteristics early in their program 
enrollment.” It included questions about their age, gender, race, and ethnicity; household 
characteristics; school experiences; and employment and income. The survey also asked respondents 
about their education and career goals; experiences with the justice and foster care systems; and their 
perceived levels of social support and self-motivation. Most of the survey’s content and wording 
mirrored that of other surveys for similar populations (e.g., the Young Parents Demonstration Survey, 
the National Longitudinal Survey of Youth, the National Longitudinal Study of Adolescent to Adult 
Health, the National Survey on Drug Use and Health, and the Re-Integration of Ex-Offenders Survey). 
It also incorporated various behavioral and mental health scales to inform matches of respondents in 
the treatment group with similar respondents in the comparison group. The Urban team developed 
the survey in fall 2015 after discussing outcomes of interest with Jobs for the Future and the program 


sites, and the survey received Institutional Review Board approval. 


Opportunity Works participants who took part in the study enrolled in the programs on a rolling 
basis between June 2015 and July 2017. The baseline survey was similarly conducted on a rolling 
basis, as participants entered programs. The survey launched in October 2015 and concluded in July 
2017. Respondents completed the web-based survey on a computer, tablet, or phone with an internet 
connection. In addition to working with staff in all sites to ensure a high response rate and improve 
the quality of the research, all enrollees in impact and comparison sites that aligned with the admission 
criteria for Opportunity Works programming were provided $10-cash incentives for completing the 


baseline survey. 


The recruitment process for survey participation proceeded as follows. First, program staff 
identified people in the treatment and comparison groups who were eligible for participation in the 
evaluation. Eligibility differed across sites by age and gender. In Hartford, enrollees ages 18 to 24 
years old were eligible to participate in the survey.2° Because Hartford treatment group members 


were largely required to have a high school credential, only people with high school credentials were 
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invited to take the survey as part of the comparison group. In Philadelphia, enrollees ages 18 to 21 
could participate. In South King County, enrollees ages 18 to 22 were eligible. These restrictions were 
based on the treatment programs’ design and were intended to minimize the differential between the 
treatment and comparison groups. Next, program staff invited those people to participate in the 
survey, commonly as part of the initial intake process. The Urban Institute provided training materials 
and a video for staff to explain the survey and the study and ensure respondents were prepared to 
give informed consent. Finally, the staff entered basic contact information for all people invited to take 
part in the survey (whether or not they agreed) in a secure document that was shared with the Urban 
Institute. This shared tracking mechanism allowed the research team to estimate response rates and 


provided additional information to inform follow-up survey outreach. 


The follow-up survey launched in December 2016 and concluded in January 2019. It was offered 
by internet, phone, and ultimately in person for nonrespondents to earlier survey efforts. The Urban 
Institute partnered with Research Support Services (RSS) to support the follow-up survey effort. The 
goal was to administer the survey 12 to 17 months after respondents completed the baseline 


survey.26 


The recruitment process for follow-up survey participation proceeded as follows. First, Urban staff 
created and shared a secure document with the names and relevant contact information, including 
primary email and phone number, of baseline survey participants with RSS. They also included a 
tracking mechanism to document and measure the number and type of efforts the research team 
made to reach each participant. Next, Urban contacted survey participants via email and text about 11 
months after they had completed their baseline survey with two goals in mind: first, to confirm that 
the contact information on file was still accurate (and record new information if that was not the case); 
and, second, to let the respondent know that they would receive a separate invitation to take the 
follow-up survey the following month.?” Urban staff then reached out to respondents again with a link 
to the web-administered follow-up survey and a reminder that they would receive $20 once they had 
completed it. If, after five contact attempts within a month, the participant did not complete the web- 
administered survey, Urban staff passed the case on to RSS to follow up via phone or in person. After 
that, RSS coordinated with Urban staff to reach out to respondents by phone or in person. Generally, 
cases were kept open for four to five more months; RSS closed a few cases because some participants 
were incarcerated. The Urban Institute supported RSS’s outreach efforts by contacting the program 


sites to gather updated contact information for any respondent who was unreachable. 28 
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SURVEY RESPONSE RATES 

It is not possible to calculate the response rate of those offered the baseline survey. The Urban 
research team asked program staff to record everyone who was offered the baseline survey, including 
those who refused, but this tracking was not sufficiently complete to accurately estimate a response 
rate. Particularly in South King County/Seattle, it was clear that not all enrollees were offered the 
baseline survey. Based on program data enrollment counts as of summer 2017, 94 percent of the 108 
reported enrollees in Hartford took the baseline survey; 74 percent of the 120 reported enrollees in 
Philadelphia took the survey; and 35 percent of the 254 reported enrollees in South King 
County/Seattle took the survey. 


Table 2 reports the response rates for the follow-up survey by site and treatment status. The 
overall response rate of 62 percent was lower than the Urban team had hoped to achieve, but this is 
the result of extensive efforts to contact this inherently mobile survey population (see endnote 28). 
Response rates among treatment group members were higher in each site and overall, likely because 
of program staff support in locating respondents and the participants’ stronger association with the 


program and the study. 


TABLE 2 
Response Rates for the Opportunity Works Follow-Up Survey 


Number of Response rate on 

Site baseline survey participants follow-up survey 
Hartford treatment 101 72% 
Hartford comparison 98 66% 
Philadelphia treatment 89 55% 
Philadelphia comparison 116 49% 
South King County treatment 89 74% 
South King County comparison 64 55% 
Overall 557 62% 
Treatment overall 279 67% 
Comparison overall 278 56% 


National Student Clearinghouse 


The National Student Clearinghouse (NSC) is a national repository of data on postsecondary 
enrollment and credential completion maintained by a nonprofit, nongovernmental organization. It 
tracks students at 3,600 colleges and universities, covering 99 percent of postsecondary students in 
the United States.2? For those who have enrolled in a postsecondary institution, the NSC provides 
data on enrollment institution and state, institution type, enrollment dates and status, class level, 


major(s) of study, graduation status and date, and degree(s) earned. 
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The Urban research team requested NSC data for all participants who completed the baseline 
survey, had a valid birthdate, and had any postsecondary activity since 2008 to also capture any pre- 
Opportunity-Works college engagement. With this data source, the research team was able to gather 
complete information on the postsecondary outcomes for nearly all study members (97 percent), even 
if they did not respond to the follow-up survey.°° As is common practice with NSC data, a nonmatch 
of a valid name-birthdate combination was taken as evidence of no college record and therefore no 


postsecondary attendance.*? 


Propensity Score Matching Approach 


To estimate a program or intervention’s effect, it is necessary to compare outcomes from those 
offered the treatment with those not offered the treatment. This comparison, however, can be biased 
if the treatment and comparison groups have very different baseline characteristics, particularly if 
those characteristics relate independently to the outcomes (e.g., if one group of students has much 
higher achievement test scores than the other). An experimental approach achieves balance between 
the groups through random assignment. In the absence of random assignment, many quasi- 


experimental approaches try to align baseline characteristics for both groups. 


The research team used propensity score matching (PSM) to estimate the Opportunity Works 
intervention’s effects. PSM is a strong approach in which a participant group and a similar comparison 
group not offered the program are pooled into one large group with a treatment status indicator. The 
analyst estimates the “propensity” to enroll in the program as a function of a number of variables that 
could affect program enrollment. The result is an equation that predicts each person’s probability of 
enrollment based on their characteristics. Then, a researcher can compare participants and 
nonparticipants with the same probability of enrollment, based on each person’s array of 
characteristics. This comparison provides a good estimate of what the participant would have 
achieved if he or she were not in the program, as long as the variables used for matching are 
exhaustive and there are not large unmeasured differences between the participating and 
nonparticipating cases. This comparison of outcomes between participants (the treatment group) and 


nonparticipants (the comparison group) produces the program impact estimates. 


The research team implemented the model in two steps, first by estimating the probability that 
each person would participate in the program based on his or her baseline characteristics—the 


“propensity” to enroll. A probit model predicted each person’s probability of program participation: 
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P; = p( Xim, Sitem), 


where X;,, was a set of baseline characteristics of person i from site m and site,, represented site 


dummies. 22 


Second, this estimated propensity score allowed the Urban team to match people in the treatment 
group to those in the comparison group with a similar likelihood of participating in the program. The 
team used a nearest-neighbor-matching-with-replacement approach, where a person in the 
comparison group could be rematched with more than one person in the treatment group. The 
statistic of interest was the difference in the average outcomes between Opportunity Works 


participants in the treatment group and the matched comparison group: 


ty = Yr p = Yor, 


Where Yr p is the average of outcome Y for Opportunity Works at follow-up and Yor is the average of 
outcome Y for people in the comparison group who matched with people in the treatment group 
based on their observable characteristics. Standard errors using the expression derived by Abadie and 
Imbens (2006) allowed the research team to test whether ¢, was statistically different from zero. After 
the matching, we test the significance of the program effect by performing a t-test for the difference 
in outcomes means between the treatment and control group, not controlling for baseline 
characteristics of people. This procedure provides consistent estimation of the treatment effect, if 
baseline covariates are balanced in both groups after matching. We also estimate standardized effect 
sizes for most of the outcomes, where the treatment effects are represented relative to the standard 
deviation of the control group’s outcome. The analysis also tests for statistical significance, taking into 
account that some domains include multiple outcomes. Statistical significance in the three domains 
that had multiple outcomes were adjusted using a procedure developed by Benjamini and Hochberg 


(1995). 


The Urban team conducted balancing tests to see if the treatment and comparison groups were 
reasonably balanced on the covariates once properly weighted using the methods above. Although it 
is never possible to tell how well the match performs on unobserved variables, the balancing analyses 
provide some indication of how confident one should be in interpreting the impact estimates. A two- 
sample t-test and a standardized bias test provided information about the balance between treatment 
and matched comparison groups. The two-sample t-test indicates if there are significant differences in 
covariate means in the treatment and comparison samples. The drawback of this approach is that it is 
sensitive to sample sizes, as the test is less likely to find significant differences in covariates when the 


sample is small. 
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Comparison Group and Matching Criteria 


The research team used the following student characteristics for the propensity score matching 


equation: 
= Personal demographics (age, race/ethnicity, gender) 
= Prior involvement with the foster care system 
= Any children 


= Educational attainment at baseline (has a high school diploma or equivalent and whether 
enrolled at a postsecondary institution before baseline) 


= Prior involvement with the justice system (ever arrested) 

= Measures of soft skills and motivation at baseline (grit scale and personal goals) 

= Public benefit recipient at the time of the baseline survey 

= Time and site of recruitment 
All matching was conducted using values of these variables measured at enrollment. These variables 
meet Caliendo and Kopeinig’s guidance that 


Only variables that influence simultaneously the participation decision and the outcome 
variable should be included...It should also be clear that only variables that are unaffected by 
participation (or the anticipation of it) should be included in the model. To ensure this, variables 
should either be fixed over time or measured before participation. In the latter case, it must be 
guaranteed that the variable has not been influenced by the anticipation of participation. 
(Caliendo and Kopeinig 2008) 
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Impact Study Findings 


This section presents the Opportunity Works impact analysis findings along a range of education, 
labor market and disconnection, and other exploratory outcomes. Opportunity Works programs’ 
effects on postsecondary enrollment or attendance were strongly positive and robust across two data 
sources: the National Student Clearinghouse (NSC) and the follow-up Opportunity Works surveys. 
The results were even stronger for young men of color, and they were consistently positive and large 
across all three impact study sites. The labor market and disconnection outcomes from the follow-up 
survey also show that program participants were substantially better off than the matched comparison 
group members, and the results were consistent for young men of color. Few additional effects 


appeared in the exploratory outcomes examined. 


The 557 respondents to the Opportunity Works baseline survey composed the study population— 
279 from the treatment group and 278 from the comparison group. Of these, 538 study participants 
provided their name and date of birth and could be matched with the NSC to measure their 
postsecondary activities. Additional outcomes data—on college applications, FAFSA completion, 
employment outcomes, goals, arrests, and measures of social functioning and support—were available 


for the 345 people who responded to both the baseline and follow-up surveys. 


The response rate for the NSC data was about 97 percent, and the response rate for the follow- 
up survey was approximately 62 percent. The high response rate for NSC makes the analysis using 
these data more credible. To assess the issue of nonresponse in the follow-up survey sample, we test 
for differences in baseline characteristics between respondents and nonrespondents to the follow-up 
survey for both the treatment and comparison groups in table A.1. The table shows that there are 
statistical differences between respondents and nonrespondents for five out of 18 the observable 
characteristics (not considering site dummies) in the treatment group and two out of 18 in the 
comparison group. Given the relatively small baseline difference between respondents and 
nonrespondents, we decided to not reweight the follow-up survey sample to adjust for nonresponse 


bias.°3 


This section also presents site-specific estimates when the sample sizes allow for precise 
estimates. The purpose of presenting site-specific estimates is to show that results are robust across 
sites. It is not advised to directly compare the results across sites because the service populations 


and intervention details differed substantially, as described earlier in this report. 


College Enrollment Outcomes from the National Student 
Clearinghouse 


There were 538 people in the Opportunity Works program and comparison group who responded to 
the baseline survey, with both names and date of birth available. These people inform the impact 
evaluation of the program on college enrollment, using NSC data (referenced as the “NSC sample’). 


Table 3 presents their distribution by site and treatment status. 


TABLE 3 
NSC Sample Sizes 
Site/group Not in sample In sample 

Total 19 538 
Total treatment 3 276 
Total comparison 16 262 
Hartford treatment 1 100 
Hartford comparison 4 94 
Philadelphia treatment 2 87 
Philadelphia comparison 10 106 
South King County treatment 0 89 
South King County comparison 2 62 


Sources: National Student Clearinghouse and Opportunity Works baseline surveys. 


Within the NSC sample, the research team compared Opportunity Works participants with 
comparison group members using a PSM approach. Effective matching balances observable baseline 
characteristics between the program participants and matched comparison group. Table B.1 
demonstrates how effectively this study matched the treatment and comparison group by comparing 
baseline characteristics. The “Treatment,” “Unmatched comparison,” and “Matched comparison” 
columns show average baseline characteristics of Opportunity Works participants, all individuals in the 
comparison group, and matched individuals in the comparison group who are used in the analysis, 
respectively. While there are substantial differences in baseline characteristics between the treatment 
and unmatched comparison groups, the characteristics of the matched comparison group are not 
statically different from the treatment group except for three characteristics. Because baseline 
covariates are balanced in both groups after matching, a simple t-test of the difference in outcomes 
between the treatment and matched comparison groups provides consistent estimation of the 


treatment effect of the program. 


Table 4 reports the impact of Opportunity Works on the outcomes for members of the treatment 
and matched comparison groups. The NSC analysis focuses on three outcomes: whether the person 


enrolled in any postsecondary institution after the baseline survey, whether the person enrolled ina 
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two-year postsecondary institution, and whether the person enrolled in a four-year postsecondary 


institution. Effects are estimated for all sites together and each site separately. 


When pooling the sites together, Opportunity Works participants are more likely to enroll in both 
two- and four-year postsecondary institutions than their matched counterparts. About 53 percent of 
program participants enrolled in a postsecondary institution after the baseline survey, while only 26 
percent did in the matched comparison group. In other words, program participation increased the 
likelihood of postsecondary enrollment by about 27 percentage points (a 104-percent increase over 
the comparison group, indicating that enrollment more than doubled). The findings demonstrate 
enrollment increases at both two-year and four-year institutions, with increases by 13 percentage 
points for each outcome (representing a 62-percent and 133-percent increase over the comparison 
group, respectively).°4 These results are not surprising, given that a much higher share of respondents 
in the treatment group received education-related services relative to the comparison group (figure 


37). 


Program participation was associated with higher postsecondary enrollment in all sites. 
Opportunity Works participants in Hartford were 25 percentage points more likely to enroll ina 
postsecondary institution than their counterparts in the matched comparison group (a 93-percent 
gain), 12 percentage points more likely in Philadelphia (a 40-percent gain), and 44 percentage points 
more likely in South King County (a 200-percent gain). Enrollment increases were concentrated in 
two-year institutions in Hartford and Philadelphia and four-year institutions in South King County. 
However, the results for four-year enrollment from South King County should be interpreted with 
caution, because many four-year institutions in the area also offer two-year degrees (e.g., Green River 
Community College). In the NSC data, institutions are classified by the highest degree they offer, and 
the data do not report the type of credential a student sought. Once again, it is important to keep in 
mind when evaluating the treatment effects across sites that each program served a population and 
offered distinct services—therefore, it is best not to directly compare their effectiveness but to take 
consistent results as reflecting the true effect of the common Back on Track postsecondary bridging 


framework. 
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TABLE 4 
Impact Estimates for Opportunity Works Participants in the NSC Sample versus the Matched 
Comparison Group 


Matched Standardized 
Outcomes __Treatment____comparison__Difference__—effectsize___ 

All Sites 

Enrolled in any postsecondary 

institution 53.3% 26.1% 0.272*** 0.62 
(0.060) 

Enrolled in a two-year 

postsecondary institution 34.1% 21.0% 0.130** 0.32 
(0.057) 

Enrolled in a four-year 

postsecondary institution 23.6% 10.1% 0.134*** 0.44 
(0.043) 

Hartford 

Enrolled in any postsecondary 

institution 52.0% 27.0% 0.250*** 0.56 
(0.067) 

Enrolled in a two-year 

postsecondary institution 45.0% 23.0% 0.220*** 0.52 
(0.062) 

Enrolled in a four-year 

postsecondary institution 11.0% 9.0% 0.020 0.07 
(0.036) 

Philadelphia 

Enrolled in any postsecondary 

institution 40.2% 28.7% 0.115* 0.25 
(0.061) 

Enrolled in a two-year 

postsecondary institution 37.9% 23.0% 0.149** 0.36 
(0.074) 

Enrolled in a four-year 

postsecondary institution 6.9% 9.2% -0.023 -0.08 
(0.039) 

South King County 

Enrolled in any postsecondary 

institution 67.4% 22.5% 0.449*** 1.08 
(0.078) 

Enrolled in a two-year 

postsecondary institution 18.0% 16.9% 0.011 0.03 

Enrolled in a four-year 

postsecondary institution 53.9% 12.4% 0.416*** 1.26 
(0.063) 


Sources: National Student Clearinghouse and Opportunity Works baseline surveys. 

Notes: Propensity score is estimated using a probit model. The matching approach is nearest neighbor with replacement. 
Standard errors are in parentheses and estimated using the expression derived in Abadie and Imbens (2006). We use t-tests 
without controlling for baseline characteristics to test statistical differences in outcomes between treatment and control groups. 
Standardized effect sizes are represented relative to the standard deviation of the control group’s outcome. *** p < 0.01, ** p < 
0.05, * p< 0.1. 
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Results for Young Men of Color 


Table 5 compares the outcomes for participants in Opportunity Works with the matched comparison 
group for young men of color. There are 159 program participants and 110 people in the unmatched 
comparison group within this demographic group.*° Overall, even stronger effects of program 
participation emerge when looking at men of color. About 52 percent of men of color from the 
Opportunity Works programs enrolled in a postsecondary institution after their baseline interview 
compared with only 8.8 percent from the matched comparison group. As a result, program 
participation is associated with a 43-percentage-point increase in college enrollment for men of color, 
a 489-percent gain. Program participation’s strong effects on enrollment also appear in both two-year 
and four-year institutions (27- and 19-percentage-point increases in the likelihood of enrollment, 


respectively). 


When looking at specific sites, program participation is associated with a 42-percentage-point 
increase in enrollment in a postsecondary institution for men of color at Hartford (a 391-percent gain), 
a 34-percentage-point increase in Philadelphia (a 696-percent gain), and a 57-percentage-point 
increase in South King County (a 664-percent gain). Once more, stronger effects appear for 
enrollment in two-year postsecondary institutions in Hartford and Philadelphia and four-year 
postsecondary institutions in South King County. Once again, it is important to keep in mind when 
evaluating the treatment effects across sites that each program serves a population and offers distinct 


services. 


TABLE 5 
Impact Estimates for Opportunity Works Participants in the NSC Sample versus the Matched 
Comparison Group—Young Men of Color 


Matched Standardized 
Outcomes Treatment comparison Difference effect size 

All Sites 

Enrolled in any postsecondary institution 52.2% 8.8% 0.434*** 1.53 
(0.064) 

Enrolled in a two-year postsecondary 

institution 35.8% 8.8% 0.270*** 0.95 
(0.057) 

Enrolled in a four-year postsecondary 

institution 18.9% 0.0% 0.189*** - 
(0.038) 

Hartford 

Enrolled in any postsecondary institution 53.0% 10.8% 0.422*** 1.36 
(0.068) 

Enrolled in a two-year postsecondary 

institution 45.8% 10.8% 0.349*** 1.12 
(0.067) 
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Matched Standardized 


Outcomes Treatment comparison Difference effect size 

Enrolled in a four-year postsecondary 

institution 12.0% 0.0% 0.120*** - 
(0.033) 

Philadelphia 

Enrolled in any postsecondary institution 39.0% 4.9% 0.341*** 1.59 
(0.083) 

Enrolled in a two-year postsecondary 

institution 39.0% 4.9% 0.341*** 1.59 
(0.083) 


Enrolled in a four-year postsecondary 
institution 0.0% 0.0% - - 


South King County 


Enrolled in any postsecondary institution 65.7% 8.6% 0.571*** 2.04 
(0.093) 

Enrolled in a two-year postsecondary 

institution 8.6% 8.6% 0.000 0.00 
(0.063) 

Enrolled in a four-year postsecondary 

institution 57.1% 0.0% 0.571*** - 
(0.074) 


Matching quality statistic 
Pseudo-R? 0.1836 


Sources: National Student Clearinghouse and Opportunity Works baseline surveys. 

Notes: Propensity score is estimated using a probit model. The matching approach is nearest neighbor with replacement. 
Standard errors are in parentheses and estimated using the expression derived in Abadie and Imbens (2006). We use t-tests 
without controlling for baseline characteristics to test statistical differences in outcomes between treatment and control groups. 
Standardized effect sizes are represented relative to the standard deviation of the control group’s outcome. *** p < 0.01, ** p< 
0.05, * p< 0.1. 


College Enrollment and Additional Outcomes from the 
Follow-Up Survey 


The follow-up survey provides insights into a broader range of outcomes for the 345 people who 
responded to both the baseline and follow-up surveys. Table 6 presents the distribution of 
respondents by site and treatment status. In addition, appendix A contains a nonresponse bias analysis 
that tests whether the 345 people who responded to the follow-up survey are statistically different 
from the 212 people in the nonresponse group by treatment group. This analysis shows that follow-up 
respondents in the treatment group were less likely to be black, less likely to be in foster care, and 
more highly educated than nonrespondents in the treatment group. In addition, respondents in the 
treatment group were more likely to take the baseline survey earlier. In the comparison group, 


responders were more likely to be older and more highly educated than nonresponders. 
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TABLE 6 
Follow-Up Survey Analysis Sample Sizes 


Site Not in sample In sample 
Total 212 345 
Total treatment 91 188 
Total comparison 121 157 
Hartford treatment 28 73 
Hartford comparison 33 65 
Philadelphia treatment 40 49 
Philadelphia comparison 59 57 
South King County treatment 23 66 
South King County comparison 29 35 


Sources: Opportunity Works baseline and follow-up surveys. 


Appendix table B.2 compares the baseline characteristics of program participants with all 
individuals in the comparison group (unmatched) and those individuals in the comparison group used 
in the analysis (matched). The goal of the table is to assess the propensity score matching quality for 
the follow-up survey sample. While there are substantial differences in baseline characteristics 
between the treatment and unmatched comparison groups, the characteristics of the matched 
comparison group are not statistically different from the treatment group except for two baseline 
characteristics. Once again, the balance in covariates in both groups after matching ensures 


consistency in our PSM estimates of the treatment effect. 


Table 7 estimates Opportunity Works program effects for three types of outcomes: education, 
labor market and disconnection, and exploratory. Exploratory outcomes are not a primarily goal of the 
program but could be affected by the intervention. Effects are estimated for all sites together in table 
7 and for each site separately in table 8. Because there are many comparisons in this analysis, the 
research team corrected for the risks associated with multiple outcomes. Treatment effects significant 
at the 10-percent level after adjusting for multiple outcomes effects are bolded, and the text notes 
where conclusions differ after multiple-outcome adjustment. Appendix C shows significance levels 
when taking into account multiple outcomes within these three domains (Benjamini and Hochberg 
1995). 


The program had several positive effects on education outcomes. Program participants were 7 
percentage points more likely to get a high school credential than people in the matched comparison 
group (an 8-percent gain over the comparison group). Opportunity Works participants were also more 


likely to apply for associate’s degrees or training programs than people in the comparison group with 
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similar baseline characteristics (14 and 7 percentage points more likely, respectively). However, there 
were no significant differences in probability of applying for a four-year degree program. Program 
participants were twice as likely to complete a Free Application for Federal Student Aid (FAFSA) 
application than the matched comparison group (71 percent versus 37 percent), an effect of 34 
percentage points. Consistent with the results from the NSC sample, Opportunity Works participants 
were 16 percentage points more likely to attend an associate’s degree program than the matched 
comparison group (an 81-percent gain) and more likely to attend any postsecondary institution (14 
percentage points).%¢ In this analysis, Opportunity Works participants were no more likely to attend a 
four-year degree program. Finally, program participants were 11 percentage points more likely than 


the matched comparison group to attend a training program (an 83-percent gain). 


The labor market and disconnection outcomes were also desirable and significant for Opportunity 
Works participants. Opportunity Works participants who were not enrolled in school were less likely 
to be looking for a job (i.e., not employed or in school and unable to find work). Only 18 percent of 
treatment group members were looking for a job, compared with 41 percent of the matched 
comparison group, with an effect of -31 percentage points (a 62-percent decrease). Program 
participants were also less likely to be disconnected from work and education. About 21 percent of 
Opportunity Works participants were not enrolled in school and not working, compared with 46 


percent of the comparison group, an effect of -24 percentage points (a 53-percent reduction). 


No significant program effects appeared for the exploratory outcomes. Opportunity Works 
participants and the comparison group members did not differ significantly in their college graduation 


goals, the likelihood of being arrested, or their social provision scores. 
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TABLE 7 
Impact Estimates for Opportunity Works Participants in the Follow-up Survey Sample versus the 
Matched Comparison Group 


Matched Standardized 
Outcome _Treatment___comparison__Difference__—effect size __ 

Educational outcomes 

High school credential 90.4% 83.5% 0.069* 0.19 
(0.038) 

Applied to an associate’s degree program 43.1% 28.7% 0.144*** 0.32 
(0.037) 

Applied to a four-year degree program 8.3% 5.5% 0.028 0.12 
(0.030) 

Applied to a training program 24.9% 17.7% 0.072*** 0.19 
(0.024) 

Completed FAFSA 70.9% 36.6% 0.343*** 0.71 
(0.074) 

Attended an associate’s degree program 36.1% 20.0% 0.161*** 0.40 
(0.047) 

Attended a four-year degree program 2.8% 3.3% -0.006 -0.03 
(0.022) 

Attended any postsecondary institution 37.8% 23.3% 0.144*** 0.34 
(0.053) 

Attended a training program 23.3% 12.8% 0.106*** 0.32 
(0.027) 

Labor market and disconnection 

outcomes 

Looking for a job 18.4% 49.1% -0.307*** -0.61 
(0.067) 

Not working and not enrolled in school 21.4% 45.8% -0.244*** -0.49 
(0.052) 

Exploratory outcomes 

Goal to graduate from a two-year 0.08 

program 62.2% 58.5% 0.037 
(0.070) 

Goal to graduate from a four-year -0.12 

program 64.4% 69.7% -0.053 
(0.042) 

Arrested since baseline 4.2% 5.4% -0.012 -0.05 
(0.025) 

Social Provision Score 3.13 3.07 0.058 0.10 
(0.061) 


Sources: Opportunity Works baseline and follow-up surveys. 

Notes: Propensity score is estimated using a probit model. The matching approach is nearest neighbor with replacement. 
Standard errors are in parentheses and estimated using the expression derived in Abadie and Imbens (2006). We use t-tests 
without controlling for baseline characteristics to test statistical differences in outcomes between treatment and control groups. 
Standardized effect sizes are represented relative to the standard deviation of the control group’s outcome. *** p < 0.01, ** p < 
0.05, * p < 0.1. Bold impacts are significant at the 10-percent level after adjusting for multiple outcomes, using the procedure 
developed by Benjamini and Hochberg (1995)—see appendix C. The looking for a job variable is only measured for those not 
enrolled in school. 
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Table 8 presents the program impact estimates by site. These results must be interpreted with 
caution because of small sample sizes at the site level (table 6). The analysis does not examine the 
likelihood of obtaining a high school credential in Hartford because virtually all program participants 
entered the program with a high school credential. Opportunity Works participants in Philadelphia 
were 28 percentage points more likely to apply to an associate’s degree program than those in the 
matched comparison group. In Hartford, participants were more likely to apply to a training program. 
Program participants in Hartford and South King County were more likely to complete a FAFSA 
application than members of the matched comparison group. Consistent with the results from the 
NSC sample, Opportunity Works participants were more likely to attend an associate’s degree 
program in Philadelphia and South King County. In Hartford, the direction of the effect on two-year 
college attendance is positive, but it is not statistically significant. No site saw a significant effect on 
four-year college attendance from the follow-up survey. Finally, program participants in Hartford were 


29 percentage points more likely to attend a training program than those in the comparison group. 


Turning to labor market and disconnection outcomes, Opportunity Works participants who were 
not in school in all sites were less likely to be looking for a job (i.e., unemployed and unable to find 
work) at the time of the follow-up interview. In addition, program participants in Hartford and South 
King County were less likely to be disconnected from work and education (20 percentage points and 
AO percentage points less likely, respectively). No significant program effects appeared for any 


exploratory outcomes in the three sites. 


TABLE 8 
Program Effects by Site for Opportunity Works Participants in the Follow-Up Survey Sample versus 
the Matched Comparison Group 


Outcome Hartford __ Philadelphia SKC 
Educational outcomes 
High school credential 7 0.122 0.000 
(0.077) (0.066) 
Applied to an associate’s degree program 0.111 0.283*** 0.079 
(0.068) (0.072) (0.092) 
Applied to a four-year degree program 0.042 0.043 0.000 
(0.045) (0.052) (0.037) 
Applied to a training program 0.181** 0.043 -0.032 
(0.078) (0.083) (0.061) 
Completed FAFSA 0.282*** 0.073 0.587*** 
(0.084) (0.109) (0.064) 
Attended an associate’s degree program 0.123 0.200** 0.177** 
(0.080) (0.078) (0.085) 
Attended a four-year degree program 0.014 -0.044 0.000 
(0.028) (0.048) (0.021) 
Attended any postsecondary institution 0.123** 0.156** 0.161** 
(0.061) (0.078) (0.076) 
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Outcome Hartford Philadelphia SKC 


Attended a training program 0.288*** -0.022 -0.016 
(0.059) (0.055) (0.058) 

Labor market and disconnection outcomes 

Looking for a job -0.212** -0.273** -0.517*** 
(0.088) (0.116) (0.082) 

Not working and not enrolled in school -0.203*** -0.077 -0.400*** 
(0.067) (0.096) (0.080) 

Exploratory outcomes 

Goal to graduate from a two-year program -0.027 0.000 0.136 
(0.081) (0.114) (0.092) 

Goal to graduate from a four-year program -0.027 -0.020 -0.106 
(0.069) (0.113) (0.073) 

Arrested since baseline -0.044 0.026 0.000 
(0.040) (0.075) (0.030) 

Social Provision Score 0.064 -0.011 0.096 
(0.113) (0.082) (0.099) 


Sources: Opportunity Works baseline and follow-up surveys. 

Notes: * No impact estimate is available for this outcome because nearly all program participants already had achieved this 
outcome at the time of enrollment. Propensity score is estimated using a probit model. The matching approach is nearest 
neighbor with replacement. Standard errors are in parentheses and estimated using the expression derived in Abadie and 
Imbens (2006). We use t-tests without controlling for baseline characteristics to test statistical differences in outcomes between 


* 


treatment and control groups. *** p < 0.01, ** p < 0.05, * p < 0.1. Bold effects are significant at the 10-percent level after 


adjusting for multiple outcomes, using the procedure developed by Benjamini and Hochberg (1995)—see appendix C. The 
looking for a job variable is only measured for those not enrolled in school. 


Results for Young Men of Color 


The final analysis examines effects for young men of color from the follow-up survey. There were 110 
Opportunity Works participants and 63 members of the comparison group who responded to both the 
baseline and follow-up surveys and were men of color.?” As shown in table 9, despite the small sample 
sizes, significant and positive program effects emerged on the likelihood of applying to an associate’s 
degree program (16 percentage points), completing FAFSA (47 percentage points), and attending an 
associate’s degree program (15 percentage points). No significant effects of program participation 
appear for labor market and disconnection outcomes for young men of color after adjusting for 
multiple outcomes, although the treatment effects are as expected. Finally, Opportunity Works young 
men of color were marginally more likely to be arrested than those in the matched comparison group, 


although the effect is no longer significant after adjusting the analysis for multiple outcomes. 
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TABLE 9 
Program Effects by Site for Opportunity Works Participants in the Follow-Up Survey Sample versus 
the Matched Comparison Group, Young Men of Color 


Matched Standardized 
Outcome Treatment comparison Difference effect size 

Educational outcomes 

High school credential 88.2% 87.3% 0.009 0.03 
(0.043) 

Applied to an associate’s degree program 36.4% 20.6% 0.159*** 0.39 
(0.035) 

Applied to a four-year degree program 10.3% 12.1% -0.019 -0.06 
(0.049) 

Applied to a training program 28.0% 20.6% 0.075 0.18 
(0.105) 

Completed FAFSA 68.6% 21.6% 0.471*** 1.14 
(0.055) 

Attended an associate’s degree program 31.5% 16.7% 0.148*** 0.40 
(0.048) 

Attended a four-year degree program 3.7% 2.8% 0.009 0.06 
(0.024) 

Attended any postsecondary institution 33.3% 19.4% 0.139*** 0.35 
(0.046) 

Attended a training program 29.6% 21.3% 0.083 0.20 
(0.095) 

Labor market and disconnection 

outcomes 

Looking for a job 19.2% 61.6% -0.425* -0.87 
(0.226) 

Not working and not enrolled in school 19.8% 23.8% -0.040 -0.09 
(0.049) 

Exploratory outcomes 

Goal to graduate from a two-year -0.22 

program 60.0% 70.0% -0.100 
(0.080) 

Goal to graduate from a four-year 0.00 

program 65.5% 65.5% 0.000 
(0.224) 

Arrested since baseline 6.2% 1.0% 0.052* 0.51 
(0.029) 

Social Provision Score 3.12 3.02 0.101 0.18 
(0.192) 


Matching quality statistic 
Pseudo-R2 0.1778 


Sources: Opportunity Works baseline and follow-up surveys. 

Notes: Propensity score is estimated using a probit model. The matching approach is nearest neighbor with replacement. 
Standard errors are in parentheses and estimated using the expression derived in Abadie and Imbens (2006). *** p < 0.01, ** p < 
0.05, * p < 0.1. We use t-tests without controlling for baseline characteristics to test statistical differences in outcomes between 
treatment and control groups. Standardized effect sizes are represented relative to the standard deviation of the control group’s 
outcome. Bold effects are significant at the 10-percent level after adjusting for multiple outcomes, using the procedure 
developed by Benjamini and Hochberg (1995)—see appendix C. The looking for a job variable is only measured for those not 
enrolled in school. 
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College Persistence 


The past section demonstrated that the Opportunity Works program had a substantial effect on 
college enrollment. This section investigates whether Opportunity Works participants stay enrolled in 


college for more than one semester. 


In the first part of this analysis, we use the data from the National Student Clearinghouse and 
restrict the sample to students in both the treatment and comparison groups who enrolled in college 
between July 1, 2015 and December 31, 2017. 38 After this restriction, the sample consists of 136 
students in the treatment group and 51 in the comparison group. Using the same matching process 
described earlier, we match college students in the treatment and comparison groups.°? We define an 
indicator variable for college persistence as equal to one if a student enrolled in college in the first or 
second semester after his or her first semester of college enrollment and zero otherwise. For example, 
if a student first enrolled in college in the 2016 second semester, the indicator for college persistence 
assumes value one if the student enrolled in college in either the first or second semester of 2017 and 
zero if the student did not enroll in college in either semester. The goal is to identify students who 


came back to college within a year after their first enrollment. 


Table 10 shows the share of college students in the treatment and matched comparison groups 
who came back to college within a year. This analysis is conditional on students enrolling. Note that 
there is lower persistence in college enrollment among Opportunity Works college enrollees relative 
to the comparison group. While 64.7 percent of Opportunity Works students came back for a second 
semester within a year of their first enrollment, 75.7 percent of students in the matched comparison 


group persisted in college for a second semester.*° 


This result may not be surprising because Opportunity Works helped more “marginal” students to 
enroll in school—those who would not have attended college otherwise. Given the program’s large 
effect on college enrollment, we might expect that these Opportunity Works students are less likely to 
persist. Conversely, comparison group members who managed to enroll in college without the benefit 
of the Back on Track supports may be more stable or motivated on their own and thus more likely able 


to persist. 
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TABLE 10 
Impact Estimates for Opportunity Works Participation on College Persistence 


Outcome Treatment Matched comparison Difference 
-0.110** 
College persistence 64.7% 75.7% (0.052) 
Sample size 136 55 


Sources: National Student Clearinghouse and Opportunity Works baseline surveys. 

Notes: Propensity score is estimated using a probit model. The matching approach is nearest neighbor with replacement. 
Standard errors are in parentheses and estimated using the expression derived in Abadie and Imbens (2006). We use t-tests 
without controlling for baseline characteristics to test statistical differences in outcomes between treatment and control groups. 
*** n < 0.01; ** p < 0.05; * p< 0.1. 


To further investigate the benefits from Opportunity Works, despite the somewhat lower 
persistence of its college enrollees, this study also evaluates the program’s effect on the likelihood of 
enrolling in college for two semesters for any participant. This second part of the analysis is conducted 
without any sample restriction on college enrollment (i.e., it is not conditional on college enrollment). 
The outcome of interest is equal to one if the program participant enrolled in college for two 
semesters (within a three-semester time frame) and zero if the participant either enrolled in college for 
only one semester or never enrolled in college. For example, the outcome two semesters of college 
enrollment would be equal to one if a student enrolled in college in the first and second semesters of 
2016 and zero for both students who enrolled in the 2016 first semester but did not enroll for the 


next two semesters and students who never enrolled in college. 


Table 11 shows that 32.4 percent of Opportunity Works participants enrolled in college for at 
least two semesters compared with 12.1 percent in the matched comparison group. This represents a 
17.3-percentage-point increase in the likelihood of two-semester college enrollment associated with 
program participation. Overall, while Opportunity Works students are less likely to persist in college 
conditional on first enrollment, Opportunity Works participants are still more likely to enroll in two 
semesters of college education than those in the matched comparison group. The intuition behind this 
result is that to be enrolled in two semesters a participant must first be enrolled in one semester. Thus, 
the analysis that does not condition on enrollment in one semester finds higher persistence for 
Opportunity Works participants, in part, because it increases the likelihood that they will enroll in one 


semester. 
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TABLE 11 
Impact Estimates for Opportunity Works Participation in Two Semesters of College Enrollment 


Outcome Treatment Matched comparison Difference 
Two semesters of 0.173*** 
college enrollment 32.4% 15.1% (0.048) 
Sample size 276 262 


Sources: National Student Clearinghouse and Opportunity Works baseline surveys. 

Notes: Propensity score is estimated using a probit model. The matching approach is nearest neighbor with replacement. 
Standard errors are in parentheses and estimated using the expression derived in Abadie and Imbens (2006). *** p < 0.01; ** p < 
0.05; * p < 0.1. 
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Discussion: Does Opportunity 
Works Help Opportunity Youth? 


For the confirmatory education impact results, it is clear that Opportunity Works has a consistent, 
large, positive effect on postsecondary enrollment. This finding is robust across data sources and 
appears even more strongly among young men of color than the program population overall. The 
magnitude of the effect is unusually large, more than doubling postsecondary enrollment overall and 
increasing it nearly six times over for the young men’s subgroup. The findings are consistent across 
the three sites in the impact study. This suggests that the Back on Track postsecondary bridging 
framework can receive credit for the positive effect, given the framework was the primary consistent 


feature across the three diverse impact sites. 


The Back on Track postsecondary bridging framework also appears to have helped participants 
reduce their chance of disconnection from education and employment. However, it did not affect 
some of the exploratory outcomes around goals for education, justice involvement, or measures of 
social functioning and support. It could be valuable to consider how to build out the elements of the 
model that might improve these outcomes, such as mentoring support, which appeared in the original 


framework but was not implemented in any Opportunity Works site (Anderson et al. 2017). 


Jobs for the Future has further refined the Back on Track model to deepen its effectiveness across 
implementation contexts, and that effort is the focus of a separate implementation evaluation. The 
lessons from this evaluation, particularly the implementations study, may serve as valuable guideposts 
to understand where refinement opportunities are and where common practices among the three 


impact sites may be contributing to such positive, consistent education results. 


This study had a relatively short follow-up period. It would be valuable to conduct future research 
with a longer follow-up period to understand how students fare in the medium- and long-term, given 
these robust short-term impacts. It would be particularly valuable to gain insight into participants’ 
persistence and success in their postsecondary pursuits and how this translates into longer-term 
economic and social well-being. It is possible that some of the exploratory outcomes would emerge 
with meaningful results in a longer time frame, given the location of many of these outcomes in the 


medium- and long-term segments of the program logic model. 


It would also be valuable to deepen the evidence base in the future with an experimental design, if 


sufficient recruitment efforts could result high demand for a program that is willing to undertake an 


experimental study. That would provide strong evidence of program effectiveness. The methods 
employed here were likely the strongest possible given the implementation constraints in this study. 
The findings are unambiguous in supporting this intervention’s effectiveness in helping opportunity 


youth access postsecondary pathways. 
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Appendix A. Nonresponse Bias 


Analysis 


TABLE A.1 


Baseline Characteristics of Study Participants, by Follow-Up Survey Response Status 


Baseline variable 
Female 
% Non-Hispanic white (alone) 
% Non-Hispanic Black (alone) 
% Hispanic and other Race 
% ages 18-19 
% ages 20-21 
% ages 22 and older 
Foster 


Presence of children 
High school diploma or 
equivalent 
NSC—enrolled ina 
postsecondary institution 
before baseline 


Ever arrested 

Mean grit (scale from 1 to 5) 
Goal: attend a two-year college 
Goal: attend a four-year 
college 

% Currently receive SNAP or 
TANF 

2015 second semester/2016 
first semester 

2016 second semester/any 
time in 2017 

Hartford 

Philadelphia 

SKC 


Number of observations 


Sources: Opportunity Works baseline and follow-up surveys. 


Treatment 
Respondent Nonrespondent 

39.6% 42.9% 

8.4% 3.6% 
50.0% 63.9%** 
41.6% 32.5% 
43.2% 47.3% 
39.6% 31.9% 
17.2% 20.9% 

9.9% 18.0%* 
19.7% 22.2% 
73.7% 70.0% 
41.6% 24.2%*** 
23.1% 23.9% 

3.65 3.75 
47.9% 47.3% 
63.8% 62.6% 
52.2% 51.9% 
48.4% 28.6%*** 
51.6% 71.4%*** 
38.8% 30.8% 
26.1% 44.0%*** 
35.1% 25.3%" 

188 91 


Comparison 
Respondent Nonrespondent 

52.9% 47.1% 
15.5% 12.2% 
56.3% 55.7% 
28.2% 32.2% 
53.4% 65.7% 
26.4% 24.8% 
20.3% 9.5%** 

8.5% 11.3% 
20.6% 16.7% 
40.3% 32.7% 
25.5% 15.2%** 
25.0% 23.7% 

3.60 3.50 
35.7% 30.6% 
52.9% 52.9% 
57.8% 57.1% 
31.2% 34.7% 
68.8% 65.3% 
41.4% 27.3% 
36.3% 48.8%** 
22.3% 24.0% 

157 121 


Note: Asterisks show statistical difference between respondents and nonrespondents. *** p < 0.01; ** p < 0.05; * p< 0.1. 
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Appendix B. Matching Quality 
NSC Sample 


The NSC sample consists of 276 Opportunity Works participants. There were 262 people in the 
unmatched comparison group and 116 retained in the matched comparison group. There are fewer 
people in the matched comparison group because the nearest-neighbor-matching-with-replacement 
approach allows for a comparison group member to match with more than one treatment group 
member if they are the best aligned on observed characteristics for multiple members of the 


treatment group. This improves matching quality and the resulting estimates. 


In table B.1, the “Treatment,” “Unmatched comparison,” and “Matched comparison” columns 
show average characteristics of Opportunity Work participants, the unmatched comparison group, 
and the matched comparison group, respectively, before the program intervention. For example, the 
share of people ages 18to 19 is 45 percent among Opportunity Work participants, 59 percent within 


the unmatched comparison group, and 41 percent within the matched comparison group. 


The column “Statistical differences” tests whether Opportunity Work participants and the 
unmatched and matched comparison groups had different baseline characteristics. Opportunity Works 
participants were less likely to be female, were older, had higher educational attainment at baseline, 
and were more likely to have a goal of attending both a two-year and four-year college than the 
unmatched group. Opportunity Works participants were also likely to take the baseline survey earlier 
in the study. As expected, the matched comparison group looks more like the treatment group in 
baseline characteristics. The only remaining significant difference between the two groups after 
matching appears in their likelihood of being in foster care, their goal of attending a four-year college 


(marginal difference), and their likelihood of being from South King County/Seattle. 


As another check, the research team reestimated the propensity score on the matched sample— 
that is, only on participants and matched nonparticipants, to compare how well the baseline 
characteristics explain the participation probability. This is measured with a statistic called a pseudo- 
R?. Table B.1 reports the pseudo-R? values of propensity score estimations before and after matching. 
After matching, there should be no systematic differences in the distribution of covariates between 
both groups and, therefore, the pseudo-R? should be low. Here, the pseudo-R? decreases from 0.1859 
before matching to 0.0596 after matching, reflecting the small residual difference between 
Opportunity Works participants and members of the matched comparison group in their baseline 


characteristics. 


TABLE B.1 
Baseline Characteristics of Opportunity Works Youth in the NSC Sample, Unmatched Comparison 
Group, and Matched Comparison Group (Averages) 


Treatment Unmatched Matched 


Baseline variable Mean Mean Mean 
Female 40.7% 50.4%** 42.2% 
% Non-Hispanic white (alone) 6.9% 14.0% 4.9% 
% Non-Hispanic Black (alone) 54.4% 55.1% 60.7% 
% Hispanic/Latinx or other race 38.7% 30.9%* 34.4% 
% ages 18-19 44.6% 58.5%*** 42.4% 
% ages 20-21 36.9% 25.7%*** 42.8% 
% ages 22 or older 18.5% 15.8% 14.8% 
Foster 12.0% 9.8% 19.8%** 
Presence of children 20.0% 18.8% 26.4% 
High school diploma or equivalent 72.6% 37.0%*** 74.1% 
Enrolled in a postsecondary institution before baseline (from NSC) 35.9% 21.4%*** 32.4% 
Ever arrested 23.2% 24.1% 22.6% 
Mean grit (scale from 1 to 5) 3.68 3.57* 3.73 
Goal: attend a two-year college 47.8% 33.2%*** 48.7% 
Goal: attend a four-year college 63.8% 54.2%** 54.9%* 
% Currently receive SNAP or TANF 52.1% 57.6% 45.0% 
2015 second semester/2016 first semester 42.0% 32.4%** 43.3% 
2016 Second semester/any time in 2017 58.0% 67.6%** 56.7% 
Hartford 36.2% 35.9% 40.7% 
Philadelphia 31.5% 40.5%** 37.5% 
South King County/Seattle 32.2% 23.7%** 21.8%** 
Matching quality statistic 
Pseudo-R2 0.1859 0.0596 
Number of observations 276 262 104 


Sources: National Student Clearinghouse and Opportunity Works baseline surveys. 
Notes: Propensity score is estimated using a probit model. The matching approach is nearest neighbor with replacement. 
Asterisks indicated statistical difference from the treatment group. *** p < 0.01; ** p < 0.05; * p< 0.1. 


Follow-Up Survey Sample 


Table B.2 compares the baseline characteristics of program participants with those of the matched 
and unmatched comparison groups to assess the quality of the propensity score matching for the 
follow-up survey sample. The matched comparison group looked more like the treatment group in 


their baseline characteristics. 


Compared with unmatched people in the comparison group, Opportunity Works participants were 
less likely to be female, less likely to be non-Hispanic white, more likely to be Hispanic/Latinx or other 
race, had higher educational attainment at baseline, and were more likely to have a goal of attending 


both a two-year and four-year college than the unmatched group. Opportunity Works participants 
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were also more likely to take the baseline survey earlier in the study and were less likely to be from 


the Philadelphia or South King County sites. 


The matched comparison group looked more like the treatment group in baseline characteristics. 
The only significant differences between the two groups remained in their likelihood of being female 
and their site of origin (marginal difference). This result supports the quality of the matching 
procedure. In addition, the pseudo-R2 showed significant improvements in the matching 
characteristics, reducing from 0.2090 before matching to 0.0682 after matching. 

TABLE B.2 


Baseline Characteristics of Opportunity Works Youth in the Follow-Up Survey Sample, Unmatched 
Comparison Group, and Matched Comparison Group (Averages) 


Treatment Unmatched Matched 


Baseline variable Mean Mean___ Mean 
Female 39.6% 52.9%** 25.7%** 
% Non-Hispanic white (alone) 8.4% 15.5%* 7.3% 
% Non-Hispanic Black (alone) 50.0% 56.3% 47.6% 
% Hispanic/Latinx or other race 41.6% 28.2%** 45.1% 
% ages 18-19 43.2% 53.4% 51.5% 
% ages 20-21 39.6% 26.4% 36.1% 
% ages 22 or older 17.2% 20.3% 12.4% 
Foster 9.9% 8.5% 9.0% 
Presence of children 19.7% 20.6% 16.2% 
High school diploma or equivalent 73.7% 40.3%*** 78.2% 
Enrolled in a postsecondary institution before baseline (from NSC) 41.6% 25.5%*** 36.1% 
Ever arrested 23.1% 25.0% 29.2% 
Mean grit (scale from 1 to 5) 3.65 3.60 3.60 
Goal: attend a two-year college 47.9% 35.7%** 42.6% 
Goal: attend a four-year college 63.8% 52.9%** 66.1% 
% Currently receive SNAP or TANF 52.2% 57.8% 51.1% 
2015 second semester/2016 first semester 48.4% 31.2%*** 38.8% 
2016 second semester/any time in 2017 51.6% 68.8%*** 61.2% 
Hartford 38.8% 41.4% 49.7%* 
Philadelphia 26.1% 36.3%** 20.8% 
South King County/Seattle 35.1% 22.3%*** 29.5% 
Matching quality statistic 
Pseudo-R2 0.2090 0.0682 
Number of observations 188 157 67 


Sources: National Student Clearinghouse and Opportunity Works baseline surveys. 
Notes: Propensity score is estimated using a probit model. The matching approach is nearest neighbor with replacement. 
Asterisks indicated statistical difference from the treatment group. *** p < 0.01; ** p < 0.05; * p < 0.1. 
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Appendix C. Multiple Outcomes 


TABLE C.1 
Significance Levels after Adjusting for Multiple Outcomes 


Men of 
Outcome All color Hartford Philadelphia SKC _ 
Educational outcomes 
High school credential . 
Applied to an associate’s degree program Mee nae aoe 
Applied to a four-year degree program 
Applied to a training program ia 7 
Completed FAFSA ae ae ere wae 
Attended an associate’s degree program bias we ~ ‘ 
Attended a four-year degree program 
Attended any postsecondary institution se “ 
Attended a training program ici ia 
Labor market and disconnection outcomes 
Looking for a job =e e _ lai 
KK 2k KK 2K K 


Not working and not enrolled in school 
Exploratory outcomes 

Goal to graduate from a two-year program 
Goal to graduate from a four-year program 
Arrested since baseline 

Social Provision Score 


Sources: Opportunity Works baseline and follow-up surveys. 

Notes: Propensity score is estimated using a probit model. The matching approach is nearest neighbor with replacement. 
Standard errors are in parentheses and estimated using the expression derived in Abadie and Imbens (2006). Statistical 
significance in the three domains that have multiple outcomes are adjusted using a procedure developed by Benjamini and 
Hochberg (1995). *** p < 0.01; ** p < 0.05; * p< 0.1. 
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Appendix D. Within-Site Match 


TABLE D.1 
Impact Estimates for Opportunity Works Participants in the NSC Sample versus the Matched 


Comparison Group—Within-Site Matching 


Outcomes Hartford Philadelphia SKC 
Enrolled in any postsecondary institution 0.360*** 0.160*** 0.553 
(0.069) (0.009) (0.508) 
Enrolled in a two-year postsecondary institution 0.320*** 0.200** 0.059 
(0.067) (0.079) (0.507) 
Enrolled in a four-year postsecondary institution 0.080** -0.080 0.541*** 
(0.036) (0.235) (0.050) 


Matching quality statistic 
Pseudo-R2 0.3088 0.2190 0.5883 


Sources: National Student Clearinghouse and Opportunity Works baseline survey. 
Notes: Propensity score is estimated using a probit model. The matching approach is nearest neighbor with replacement within 


site. *** p < 0.01; ** p < 0.05; * p < 0.1. 
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Notes 
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“Table 219.75. Percentage of High School Dropouts among Persons 16 to 24 Years Old (Status Dropout Rate), 
by Income Level, and Percentage Distribution of Status Dropouts, by Labor Force Status and Years of School 
Completed: Selected Years, 1970 through 2016,” US Department of Education, last updated 2017, 
https://nces.ed.gov/programs/digest/d17/tables/dt17_219.75.asp?current=yes. 


“Table 219.80. Percentage of High School Dropouts among Persons 16 to 24 Years Old (Status Dropout Rate) 
and Number of Status Dropouts, by Noninstitutionalized or Institutionalized Status, Birth in or Outside of the 
United States, and Selected Characteristics: Selected Years, 2006 through 2016,” US Department of Education. 
last updated 2017, https://nces.ed.gov/programs/digest/d17/tables/dt17_219.80.asp?current=yes. 


“Table 219.75. Percentage of High School Dropouts among Persons 16 to 24 Years Old (Status Dropout Rate), 
by Income Level, and Percentage Distribution of Status Dropouts, by Labor Force Status and Years of School 
Completed: Selected Years, 1970 through 2016.” 


“Table 219.75. Percentage of High School Dropouts among Persons 16 to 24 Years Old (Status Dropout Rate), 
by Income Level, and Percentage Distribution of Status Dropouts, by Labor Force Status and Years of School 
Completed: Selected Years, 1970 through 2016.” 


“Table 219.80. Percentage of High School Dropouts among Persons 16 to 24 Years Old (Status Dropout Rate) 
and Number of Status Dropouts, by Noninstitutionalized or Institutionalized Status, Birth in or Outside of the 
United States, and Selected Characteristics: Selected Years, 2006 through 2016.” 


“Table 219.75. Percentage of High School Dropouts among Persons 16 to 24 Years Old (Status Dropout Rate), 
by Income Level, and Percentage Distribution of Status Dropouts, by Labor Force Status and Years of School 
Completed: Selected Years, 1970 through 2016.” 


“Employment, Wages, and Projected Change in Employment by Typical Entry-Level Education,” US Department 
of Labor, last updated April 12, 2091, https://www.bls.gov/emp/tables/education-summary.htm. 


As shorthand and in recognition of the primary short-term goal of postsecondary enrollment, the report will 
refer to “postsecondary/career bridging” as “postsecondary bridging.” 


See CNCS (2014) for a description of the evaluation requirements for SIF grants. 


A “site” in this report refers to the backbone organization and collection of partners implementing Opportunity 
Works in each city. See the site profiles in the next chapter for a list of partner organizations in each site. 


Note that this is as of the latest program deliverable; it is possible participants engaged longer, but this was not 
observed by the end of the data coverage period. 


Note that this does not include the most recent two quarters of program data, as data was recorded differently 
across quarters. Obtaining this information may be possible but was not feasible at the time of writing in May 
2019. 


Note that this does not include the most recent two quarters of program data, as data was recorded differently 
across quarters. Obtaining this information may be possible but was not feasible at the time of writing in May 
2019. 


The program data do not define “regular contact” explicitly, but the data files do note that synonyms for this 
status are “strong rapport” and “strong comfort level.” 


The navigators were stretched thin and participants were often not on site to receive these services. 
Navigators often had to travel to participants’ homes to meet them, with one estimating they spent 25 percent 
of their time at work driving. 


16 “Graduated” indicates that a student completed their bridging program and enrolled in a postsecondary 
program. 


17 Bard seminar is included in dual enrollment 


18 Note that 40 percent of participants in South King County/Seattle already had a high school credential at 
program enrollment, so this outcome would not be relevant to them. 


19 For an in-depth discussion of fidelity to the Back on Track model, see Anderson et al. (2017). 


20 The original evaluation plan also asked about the following longer-term outcomes: earn college credits, enroll 
in college for more than one semester, enroll in college for more than one year, earn industry-recognized 
credentials, earn college credentials, and complete 12 credits of college coursework. It also asked if 
respondents were more likely to be college ready, but this was not feasible to measure consistently beyond 
observing college applications and attendance. 


21 The original evaluation plan also asked about the following longer-term outcomes: more likely to become 
employed in quality jobs based on wages, hours worked (full/part time), benefits, and likely to earn more than 
the comparison group. 


22 The original wording of this question asked if treatment group members were more likely to become employed, 
but the research team determined that unemployment (among those not in school) was a better measure for if 
people were meeting their employment goals. 


23 The original research question asked separately about people of different racial/ethnic groups (e.g., African 
Americans and Latinx people), different ages, and different gender categories. Because of the explicit focus on 
young men of color in the Opportunity Works initiative, recognition of intersectional identities’ importance in 
determining life outcomes, sample size restrictions, and concerns about increasing the number of multiple 
comparisons, the research team simplified these categories into an analysis focused on young men of color. 


24 Treatment group members took the baseline survey as soon after enrollment as possible. This was generally 
within the first week of program participation, though there was some variation in individual cases, particularly 
for early program enrollees in Hartford. In Hartford, the average delay between enrollment and survey 
completion was 73 days, with a range from O to 593 days. In Philadelphia, the average delay was 17 days, with 
a range from 0 to 113 days. In South King County, the average delay was 56 days with a range from O to 258 
days. 


Where possible, comparison programs also administered the survey early in comparison group members’ 
enrollment. However, some members of the comparison group took the survey at varying times relative to 
their first engagement with services provided by comparison programs. One reason for this discrepancy is that 
several comparison programs were ongoing, meaning there was not a “starting point” that was analogous to 
the treatment programs. In the American Job Center comparison program in Hartford, comparison group 
respondents enrolled in the study and took the baseline survey after they had left the comparison organization. 


25 Only males ages 18- to 24-years-old were eligible to participate in the survey initially, but because of changes 
in the site’s recruitment policies, females also became eligible for the study in early 2017. 


26 Treatment group members took the follow-up survey as soon after a year post-baseline survey as possible. 
This was generally within 12 to 17 months, though there was some variation in individual cases. In Hartford, 
the average gap between baseline and follow-up survey completion was 14 months, with a range from 11 to 
18 months. In Philadelphia, the average gap was also 14 months, with a range from 11 to 19 months. In South 
King County, the average gap was 13 months, with a range from 11 to 18 months. 


27 Beginning in February 2017, the Urban team contacted respondents on a quarterly basis after baseline survey 
completion to maintain more regular communication and update contact information to improve survey 
responses when the respondents became eligible for the follow-up survey. 
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The Urban team and RSS also attempted other strategies to boost response rates, including increasing the 
incentive amount to $30 for refusals, offering a $10 “finder’s fee” for family members that helped RSS survey 
fielders locate respondents, conducting address searches through online sources, and sending postcards by 
mail to unreachable respondents. 


“About the Clearinghouse,” National Student Clearinghouse, https://studentclearinghouse.org/about/. 
A small subset of students did not have valid birthdates and could not be submitted to the NSC for matching. 


The NSC’s matching algorithm is proprietary, but it involves probabilistic matching that reportedly has a very 
high accuracy rate. Additional information about matching procedures and the interpretation of nonmatches 
appears in NSC (2017b). 


This strategy leads to the possibility that treatment group members in one site could be matched with 
comparison group members in another site. While this could cause bias if unobserved site-specific differences 
are important, it has the advantage of allowing for a closer match between treatment and comparison group 
members. 


A potential issue with the NSC data is that it might classify students taking reengagement high school courses 
in community college as enrolled in a postsecondary institution. This misclassification of college enrollment 
likely affects students in treatment and comparison groups and is potentially netted out when we estimate 
program effects. 


When estimating within site effects, the analysis allows program participants to match with people in 
comparison groups from other sites. For example, an Opportunity Works participant in Philadelphia can be 
matched with a comparison group member from Hartford, if they share most baseline characteristics. As a 
robustness check, the research team estimated treatment effect restricting matches within the same site in 
appendix D. Overall, program effects were stronger (more positive) when constraining to within-site matching, 
but the matching quality was much worse, and therefore those results are not preferred and not presented as 
the primary results in this report. 


The matching quality for the men of color is poorer than for the overall population (Pseudo-R2 equals 0.18). 
Therefore, the estimated program effects for this group are suggestive rather than conclusive. 


The NSC and the survey results for “any postsecondary” are both large and positive, but the magnitudes differ, 
with larger impact estimates based on data from the NSC (0.272 percentage points) relative to the survey 
(0.144 percentage points). This difference is expected because of nonresponse in the follow-up survey. If 
respondents to the follow-up survey were more stable or responsive, as suggested by the bias estimates in 
appendix A, then they might be expected to attend postsecondary school at higher rates than nonrespondents, 
regardless of program interventions. Therefore, it would be reasonable for the program’s effect to be smaller 
for this group. Because the NSC results cover a near-census of participants in the study, those are the 
preferred findings for postsecondary enrollment/attendance. 


The matching quality for the men of color is poorer than for the overall population (Pseudo-R2 equals 0.177). 
Therefore, the estimated program effects for this group are suggestive rather than conclusive. 


A student is defined as enrolled in college if he or she is enrolled for 30 days or more in a semester. 


This analysis is exploratory because the matching between college enrollees in the treatment and comparison 
groups within is poor (Pseudo-R2 equals 0.20)—primarily because of smaller sample size. 


As a reference, the overall persistence rate for students starting out in fall 2015 in two-year colleges was 63 
percent. It was 70 percent for full-time students, 57 percent for part-time students, and 58 percent for non- 
degree-seeking students (National Student Clearinghouse 2017) 
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